Get ready for release python-2.4-

Need print without parens. Handle old-style classes more properly?

.gitignore

@@ -17,4 +17,5 @@
 __pycache__
 build
 /.venv*
-/.idea
+/.idea
+/.hypothesis

.travis.yml

@@ -3,16 +3,10 @@ language: python
 sudo: false
 
 python:
-  - '3.5'
-  - '2.7.12'
-  - '2.6'
-  - '3.3'
-  - '3.4'
-  - '3.2'
-  - '3.6'
+  - '2.7' # this is a cheat here because travis doesn't do 2.4-2.6
 
 install:
-- pip install -r requirements.txt
+- pip install -e .
 - pip install -r requirements-dev.txt
 
 script:

HISTORY.md

@@ -44,8 +44,8 @@ it appears that Hartmut did most of the work to get this code to
 accept the full Python language. He added precedence to the table
 specifiers, support for multiple versions of Python, the
 pretty-printing of docstrings, lists, and hashes. He also wrote test and verification routines of
-deparsed bytecode, and used this in an extensive set of tests that he also wrote. He says he could verify against the 
-entire Python library. However I have subsequently found small and relatively obscure bugs in the decompilation code. 
+deparsed bytecode, and used this in an extensive set of tests that he also wrote. He says he could verify against the
+entire Python library. However I have subsequently found small and relatively obscure bugs in the decompilation code.
 
 decompyle2.2 was packaged for Debian (sarge) by
 [Ben Burton around 2002](https://packages.qa.debian.org/d/decompyle.html). As
@@ -66,7 +66,7 @@ code to handle first Python 2.3 and then 2.4 bytecodes. Because of
 jump optimization introduced in the CPython bytecode compiler at that
 time, various JUMP instructions were classifed as going backwards, and
 COME FROM instructions were reintroduced.  See
-[RELEASE-2.4-CHANGELOG.txt](https://github.com/rocky/python-uncompyle6/blob/master/DECOMPYLE-2.4-CHANGELOG.txt) 
+[RELEASE-2.4-CHANGELOG.txt](https://github.com/rocky/python-uncompyle6/blob/master/DECOMPYLE-2.4-CHANGELOG.txt)
 for more details here. There wasn't a public
 release of RELEASE-2.4 and bytecodes other than Python 2.4 weren't
 supported. Dan says the Python 2.3 version could verify the entire
@@ -98,9 +98,20 @@ so. Then hamled made a few commits earler on, while Eike Siewertsen
 made a few commits later on. But mostly wibiti, and Guenther
 Starnberger got the code to where uncompyle2 was around 2012.
 
-In `uncompyle`, decompilation of python bytecode 2.5 & 2.6 is done by
-transforming the byte code into a a pseudo 2.7 python bytecode and is
-based on code from Eloi Vanderbeken.
+While John Aycock and Hartmut Goebel were well versed in compiler
+technology, those that have come afterwards don't seem to have been as
+facile in it.  Furthermore, documentation or guidance on how the
+decompiler code worked, comparison to a conventional compiler
+pipeline, how to add new constructs, or debug grammars was weak. Some
+of the grammar tracing and error reporting was a bit weak as well.
+
+Given this, perhaps it is not surprising that subsequent changes
+tended to shy away from using the built-in compiler technology
+mechanisms and addressed problems and extensions by some other means.
+
+Specifically, in `uncompyle`, decompilation of python bytecode 2.5 & 2.6
+is done by transforming the byte code into a pseudo-2.7 Python
+bytecode and is based on code from Eloi Vanderbeken.
 
 This project, `uncompyle6`, abandons that approach for various
 reasons. However the main reason is that we need offsets in fragment
@@ -120,10 +131,10 @@ while, handling Python bytecodes from Python versions 2.5+ and
 3.2+. In doing so, it has been expedient to separate this into three
 projects:
 
-* bytecode loading and disassembly ([xdis](https://pypi.python.org/pypi/xdis)),
+* marshaling/unmarshaling, bytecode loading and disassembly ([xdis](https://pypi.python.org/pypi/xdis)),
 * parsing and tree building ([spark_parser](https://pypi.python.org/pypi/spark_parser)),
 * this project - grammar and semantic actions for decompiling
-  ([uncompyle6](https://pypi.python.org/pypi/spark_parser)).
+  ([uncompyle6](https://pypi.python.org/pypi/uncompyle6)).
 
 
 Over the many years, code styles and Python features have
@@ -162,5 +173,8 @@ support has been lagging.
 Tests for the project have been, or are being, culled from all of the
 projects mentioned.
 
+For a little bit of the history of changes to the Early-algorithm parser,
+see the file [NEW-FEATURES.rst](https://github.com/rocky/python-spark/blob/master/NEW-FEATURES.rst) in the [python-spark github repository](https://github.com/rocky/python-spark).
+
 NB. If you find mistakes, want corrections, or want your name added
 (or removed), please contact me.

HOW-TO-REPORT-A-BUG.md

@@ -2,32 +2,120 @@
 
 ## The difficulty of the problem
 
-There is no Python decompiler yet, that I know about that will
-decompyle everything. This one probably does the
-best job of *any* Python decompiler. But it is a constant work in progress: Python keeps changing, and so does its code generation.
+This decompiler is a constant work in progress: Python keeps
+changing, and so does its code generation.
 
-I have found bugs in *every* Python decompiler I have tried. Even
-those where authors/maintainers claim that they have used it on
-the entire Python standard library. And I don't mean that
-the program doesn't come out with the same Python source instructions,
-but that the program is *semantically* not equivalent.
+There is no Python decompiler yet that I know about that will
+decompile everything. Overall, I think this one probably does the best
+job of *any* Python decompiler that handles such a wide range of
+versions.
 
-So it is likely you'll find a mistranslation in decompiling.
+But at any given time, there are maybe dozens of valid Python bytecode
+files that I know of that will cause problems. And when I get through
+those and all the issues of decompiler bugs that are currently logged,
+I could probably easily find dozens more bugs just by doing a
+decompile of all the Python bytecode on any one of my
+computers. Unless you want to help out by _fixing_ bugs, or are
+willing to do work by isolating and narrowing bugs, don't feel you are
+doing me a favor by doing scans on your favorite sets of bytecode
+files.
+
+In sum, it is not uncommon that you will find a mistranslation in
+decompiling. Furthermore, you may be expected to do some work in order
+to have your bug worthy of being considered above other bugs.
+
+No one is getting paid to work to work on this project, let alone bugs
+you may have an interest in. If you require decompiling bytecode
+immediately, consider using a decompilation service.
+
+## Is it really a bug?
+
+
+### Do you have valid bytecode?
+
+As mentioned in README.rst, this project doesn't handle obfuscated
+code. See README.rst for suggestions for how to remove some kinds of
+obfuscation.
+
+Checking if bytecode is valid is pretty simple: disassemble the code.
+Python comes with a disassembly module called `dis`. A prerequisite
+module for this package, `xdis` has a cross-python version
+disassembler.
+
+### Semantic equivalence vs. exact source code
+
+Almost all versions of Python can perform some sort of code
+improvement that can't be undone. In earlier versions of Python it is
+rare; in later Python versions, it is more common.
+
+If the code emitted is semantically equivalent, then this isn't a bug.
+
+
+For example the code might be
+
+```
+if a:
+  if b:
+     x = 1
+```
+
+and we might produce:
+
+```
+if a and b:
+  x = 1
+```
+
+These are equivalent. Sometimes
+
+```
+else:
+   if ...
+
+```
+
+may out as `elif`.
+
+
+As mentioned in the README. It is possible that Python changes what
+you write to be more efficient. For example, for:
+
+
+```
+if True:
+  x = 5
+```
+
+Python will generate code like:
+
+```
+x = 5
+```
+
+So just because the text isn't the same, does not
+necessarily mean there's a bug.
 
 ## What to send (minimum requirements)
 
 The basic requirement is pretty simple:
 
 * Python bytecode
-* Source text
+* Python source text
+
+Please don't put files on download services that one has to register
+for or can't get to by issuing a simple `curl` or `wget`. If you can't
+attach it to the issue, or create a github gist, then the code you are
+sending is too large.
+
+Also try to narrow the bug. See below.
 
 ## What to send (additional helpful information)
 
 Some kind folks also give the invocation they used and the output
-which usually includes an error message produced. This is helpful. I
-can figure out what OS you are running this on and what version of
-*uncomplye6* was used. Therefore, if you don't provide the input
-command and the output from that, please give:
+which usually includes an error message produced. This is
+helpful. From this, I can figure out what OS you are running this on
+and what version of *uncomplye6* was used. Therefore, if you don't
+provide the input command and the output from that, please give:
 
 * _uncompyle6_ version used
 * OS that you used this on
@@ -48,11 +136,17 @@ Well, you could learn. No one is born into this world knowing how to
 disassemble Python bytecode. And as Richard Feynman once said, "What
 one fool can learn, so can another."
 
+If this is too difficult, or too time consuming, or not of interest to
+you, then perhaps what require is a decompilation service. [Crazy
+Compilers](http://www.crazy-compilers.com/decompyle/) offers a
+byte-code decompiler service for versions of Python up to 2.6. (If
+there are others around let me know and I'll list them here.)
+
 ## Narrowing the problem
 
-I don't need the entire source code base for which one file or module
-can't be decompiled. I just need that one file or module only. If
-there are several files, file a bug report for each file.
+I don't need or want the entire source code base for which one file or
+module can't be decompiled. I just need those file(s) or module(s).
+If there are several files, file a bug report for each file.
 
 Python modules can get quite large, and usually decompilation problems
 occur in a single function or maybe the main-line code but not any of
@@ -66,3 +160,13 @@ properly on a neighboring version of Python. That is helpful too.
 
 In sum, the more you can isolate or narrow the problem, the more
 likley the problem will be fixed and fixed sooner.
+
+## Confidentiality of Bug Reports
+
+When you report a bug, you are giving up confidentiality to the source
+code and the byte code. However, I would imagine that if you have
+narrowed the problem sufficiently, confidentiality little that
+remains would not be an issue.
+
+However feel free to remove any commments, and modify variable names
+or constants in the source code.

Makefile

@@ -11,7 +11,7 @@ RM      ?= rm
 LINT    = flake8
 
 #EXTRA_DIST=ipython/ipy_trepan.py trepan
-PHONY=all check clean pytest check-long dist distclean lint flake8 test rmChangeLog clean_pyc
+PHONY=all check clean distcheck pytest check-long dist distclean lint flake8 test rmChangeLog clean_pyc
 
 TEST_TYPES=check-long check-short check-2.7 check-3.4
 
@@ -36,13 +36,15 @@ check-2.7 check-3.3 check-3.4: pytest
 check-3.0 check-3.1 check-3.2 check-3.5 check-3.6:
 	$(MAKE) -C test $@
 
+check-3.7: pytest
+
 #:Tests for Python 2.6 (doesn't have pytest)
-check-2.6:
+check-2.4 check-2.5 check-2.6:
 	$(MAKE) -C test $@
 
-#:PyPy 2.6.1 or PyPy 5.0.1
+#:PyPy 2.6.1 PyPy 5.0.1, or PyPy 5.8.0-beta0
 # Skip for now
-2.6 5.0 5.3:
+2.6 5.0 5.3 5.6 5.8:
 
 #:PyPy pypy3-2.4.0 Python 3:
 pypy-3.2 2.4:
@@ -58,9 +60,13 @@ clean: clean_pyc
 	(cd test && $(MAKE) clean)
 
 #: Create source (tarball) and wheel distribution
-dist:
+dist: distcheck
 	$(PYTHON) ./setup.py sdist bdist_wheel
 
+# perform some checks on the package via setup.py
+distcheck:
+	$(PYTHON) ./setup.py check
+
 #: Remove .pyc files
 clean_pyc:
 	( cd uncompyle6 && $(RM) -f *.pyc */*.pyc )
@@ -87,7 +93,7 @@ bdist_egg:
 
 
 #: Create binary wheel distribution
-bdist_wheel:
+wheel:
 	$(PYTHON) ./setup.py bdist_wheel
 
 

NEWS

@@ -1,7 +1,111 @@
-uncompyle6 2.10.0 2016-05-30 Elaine Gordon
+uncompyle6 2.13.3 2017-11-13
 
-- Add fuzzy offset deparse lookup
-- 3.6 bugfixes
+Overall: better 3.6 decompiling and some much needed code refactoring and cleanup
+
+
+- Start noting names in for template-action names; these are
+  used to check/assert we have the right node type
+- Simplify <import_from> rule
+- Pypy 5.80-beta testing tolerance
+- Start to clean up instruction mangling phase by using 3.6-style instructions
+  rather trying to parse the bytecode array. This largely been done in for versions 3.x;
+  3.0 custom mangling code has been reduced;
+  some 2.x conversion has been done, but more is desired. This make it possible to...
+- Handle EXTENDED_ARGS better. While relevant to all Python versions it is most noticeable in
+  version 3.6+ where in switching to wordcodes the size of operands has been reduced from 2**16
+  to 2**8. JUMP instruction then often need EXTENDED_ARGS.
+- Refactor find_jump_targets() with via working of of instructions rather the bytecode array.
+- use --weak-verify more and additional fuzzing on verify()
+- fragment parser now ignores errors in nested function definitions; an parameter was
+  added to assist here. Ignoring errors may be okay because the fragment parser often just needs,
+  well, *fragments*.
+- Distinguish RETURN_VALUE from RETURN_END_IF in exception bodies better in 3.6
+- bug in 3.x language changes: import queue va import Queue
+- reinstate some bytecode tests since decompiling has gotten better
+- Revise how to report a bug
+
+uncompyle6 2.13.2 2017-10-12
+
+- Re-release using a more automated approach
+
+uncompyle6 2.13.1 2017-10-11
+
+- Re-release because Python 2.4 source uploaded rather than 2.6-3.6
+
+uncompyle6 2.13.0 2017-10-10
+
+- Fixes in deparsing lambda expressions
+- Improve table-semantics descriptions
+- Document hacky customize arg count better (until we can remove it)
+- Update to use xdis 3.7.0 or greater
+
+uncompyle6 2.12.0 2017-09-26
+
+- Use xdis 3.6.0 or greater now
+- Small semantic table cleanups
+- Python 3.4's terms a little names better
+- Slightly more Python 3.7, but still failing a lot
+- Cross Python 2/3 compatibility with annotation arguments
+
+uncompyle6 2.11.5 2017-08-31
+
+- Skeletal support for Python 3.7
+
+uncompyle6 2.11.4 2017-08-15
+
+* scanner and parser now allow 3-part version string lookups,
+  e.g. 2.7.1 We allow a float here, but if passed a string like '2.7'. or
+* unpin 3.5.1. xdis 3.5.4 has been releasd and fixes the problems we had. Use that.
+* some routnes here moved to xdis. Use the xdis version
+* README.rst: Link typo Name is trepan2 now not trepan
+* xdis-forced change adjust for COMPARE_OP "is-not" in
+  semanatic routines. We need "is not".
+* Some PyPy tolerance in validate testing.
+* Some pyston tolerance
+
+uncompyle6 2.11.3 2017-08-09
+
+Very minor changes
+
+- RsT doc fixes and updates
+- use newer xdis, but not too new; 3.5.2 breaks uncompyle6
+- use xdis opcode sets
+- xdis "exception match" is now "exception-match"
+
+uncompyle6 2.11.2 2017-07-09
+
+- Start supporting Pypy 3.5 (5.7.1-beta)
+- use xdis 3.5.0's opcode sets and require xdis 3.5.0
+- Correct some Python 2.4-2.6 loop detection
+- guard against badly formatted bytecode
+
+uncompyle6 2.11.1 2017-06-25
+
+- Python 3.x annotation and function signature fixes
+- Bump xdis version
+- Small pysource bug fixes
+
+uncompyle6 2.11.0 2017-06-18 Fleetwood
+- Major improvements in fragment tracking
+  * Add nonterminal node in extractInfo
+  * tag more offsets in expressions
+  * tag array subscripts
+  * set YIELD value offset in a <yield> expr
+  * fix a long-standing bug in not adjusting final AST when melding other deparse ASTs
+- Fixes yet again for make_function node handling; document what's up here
+- Fix bug in snowflake Python 3.5 *args kwargs
+
+uncompyle6 2.10.1 2017-06-3 Marylin Frankel
+
+- fix some fragments parsing bugs
+  - was returning the wrong type sometimes in deparse_code_around_offset()
+  - capture function name in offsets
+  - track changes to ifelstrmtr node from pysource into fragments
+
+uncompyle6 2.10.0 2017-05-30 Elaine Gordon
+
+- Add fuzzy offset deparse look up
+- 3.6 bug fixes
    - fix EXTENDED_ARGS handling (and in 2.6 and others)
    - semantic routine make_function fragments.py
    - MAKE_FUNCTION handling
@@ -12,19 +116,19 @@ uncompyle6 2.10.0 2016-05-30 Elaine Gordon
 - 3.5 FUNCTION_VAR bug
 - 3.x pass statement insdie while True
 - Improve 3.2 decompilation
-- Fixed -o argument processing (Gregrory)
+- Fixed -o argument processing (grkov90)
 - Reduce scope of LOAD_ASSERT as expr to 3.4+
 - "await" statement fixes
 - 2.3, 2.4 "if 1 .." fixes
 - 3.x annotation fixes
 
-uncompyle6 2.9.11 2016-04-06
+uncompyle6 2.9.11 2017-04-06
 
 - Better support for Python 3.5+ BUILD_MAP_UNPACK
 - Start 3.6 CALL_FUNCTION_EX support
 - Many decompilation bug fixes. (Many more remain). See ChangeLog
 
-uncompyle6 2.9.10 2016-02-25
+uncompyle6 2.9.10 2017-02-25
 
 - Python grammar rule fixes
 - Add ability to get grammar coverage on runs
@@ -91,7 +195,7 @@ uncompyle6 2.9.6 2016-11-20
 uncompyle6 2.9.5 2016-11-13
 
 - Fix Python 3 bugs:
-  * improprer while 1 else
+  * improper while 1 else
   * docstring indent
   * 3.3 default values in lambda expressions
   * start 3.0 decompilation (needs newer xdis)
@@ -101,12 +205,12 @@ uncompyle6 2.9.5 2016-11-13
 uncompyle6 2.9.4 2016-11-02
 
 - Handle Python 3.x function annotations
-- track def keywoard-parameter line-splitting in source code better
+- track def keyword-parameter line-splitting in source code better
 - bump min xdis version to mask previous xdis bug
 
 uncompyle6 2.9.3 2016-10-26
 
-Release forced by incompatiblity change in xdis 3.2.0.
+Release forced by incompatibility change in xdis 3.2.0.
 
 - Python 3.1 bugs:
   * handle "with ... as"
@@ -138,7 +242,7 @@ uncompyle6 2.9.0 2016-10-09
   this Forces change in requirements.txt and _pkg_info_.py
 - Start Python 1.5 decompiling; another round of work is needed to
   remove bugs
-- Simpify python 2.1 grammar
+- Simplify python 2.1 grammar
 - Fix bug with -t ...  Wasn't showing source text when -t option was given
 - Fix 2.1-2.6 bug in list comprehension
 
@@ -161,7 +265,7 @@ control-flow structure detection is done.
     . 3.0 .. 3.2 *args processing
     . 3.0 .. 3.2 call name and kwargs bug
     . 3.0 .. getting parameter of *
-    . 3.0 .. handling varible number of args
+    . 3.0 .. handling variable number of args
     . 3.0 .. "if" structure bugs
   * 3.5+ if/else bugs
   * 2.2-2.6 bugs
@@ -212,7 +316,7 @@ uncompyle6 2.7.1 2016-07-26
 
 uncompyle6 2.7.0 2016-07-15
 
-- Many Syntax and verifification bugs removed
+- Many Syntax and verification bugs removed
   tested on standard libraries from 2.3.7 to 3.5.1
   and they all decompile and verify fine.
   I'm sure there are more bugs though.
@@ -239,9 +343,9 @@ uncompyle6 2.6.0 2016-07-07
 - Better <2.6 vs. 2.7 grammar separation
 - Fix some 2.7 deparsing bugs
 - Fix bug in installing uncompyle6 script
-- Doc improvments
+- Doc improvements
 
-uncompyle6 2.5.0 2016-06-22  Summer Solstace
+uncompyle6 2.5.0 2016-06-22  Summer Solstice
 
 - Much better Python 3.2-3.5 coverage.
   3.4.6 is probably the best;3.2 and 3.5 are weaker
@@ -253,7 +357,7 @@ uncompyle6 2.5.0 2016-06-22  Summer Solstace
 uncompyle6 2.4.0 2016-05-18 (in memory of Lewis Bernstein)
 
 - Many Python 3 bugs fixed:
-  * Python 3.2 to 3.5 libaries largely
+  * Python 3.2 to 3.5 libraries largely
     uncompyle and most verify
 - pydisassembler:
   * disassembles all code objects in a file
@@ -311,7 +415,7 @@ uncompyle6 2.2.0 2016-04-30
 
 uncompyle6 2.2.0 2016-04-02
 
-- Support single-mode (in addtion to exec-mode) compilation
+- Support single-mode (in addition to exec-mode) compilation
 - Start to DRY Python 2 and Python 3 grammars
 - Fix bug in if else ternary  construct
 - Fix bug in uncomplye6 -d and -r options (via lelicopter)

README.rst

@@ -1,10 +1,10 @@
-|buildstatus| |Supported Python Versions|
+|buildstatus|
 
 uncompyle6
 ==========
 
 A native Python cross-version Decompiler and Fragment Decompiler.
-Follows in the tradition of decompyle, uncompyle, and uncompyle2.
+The successor to decompyle, uncompyle, and uncompyle2.
 
 
 Introduction
@@ -12,7 +12,7 @@ Introduction
 
 *uncompyle6* translates Python bytecode back into equivalent Python
 source code. It accepts bytecodes from Python version 1.5, and 2.1 to
-3.6 or so, including PyPy bytecode and Dropbox's Python 2.5 bytecode.
+3.7 or so, including PyPy bytecode and Dropbox's Python 2.5 bytecode.
 
 Why this?
 ---------
@@ -56,7 +56,7 @@ This uses setup.py, so it follows the standard Python routine:
 
 ::
 
-    pip install -r requirements.txt
+    pip install -e .
     pip install -r requirements-dev.txt
     python setup.py install # may need sudo
     # or if you have pyenv:
@@ -171,9 +171,12 @@ See Also
 * https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique than what is used here.
 * https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Include some fixes like supporting function annotations
 * The HISTORY_ file.
+* `How to report a bug <https://github.com/rocky/python-uncompyle6/blob/master/HOW-TO-REPORT-A-BUG.md>`_
+* https://github.com/rocky/python-xdis : Cross Python version disassembler
+* https://github.com/rocky/python-xasm : Cross Python version assembler
 
-.. |downloads| image:: https://img.shields.io/pypi/dd/uncompyle6.svg
-.. _trepan: https://pypi.python.org/pypi/trepan
+
+.. _trepan: https://pypi.python.org/pypi/trepan2
 .. _HISTORY: https://github.com/rocky/python-uncompyle6/blob/master/HISTORY.md
 .. _debuggers: https://pypi.python.org/pypi/trepan3k
 .. _remake: https://bashdb.sf.net/remake
@@ -181,7 +184,5 @@ See Also
 .. _this: https://github.com/rocky/python-uncompyle6/wiki/Deparsing-technology-and-its-use-in-exact-location-reporting
 .. |buildstatus| image:: https://travis-ci.org/rocky/python-uncompyle6.svg
 		 :target: https://travis-ci.org/rocky/python-uncompyle6
-.. |Supported Python Versions| image:: https://img.shields.io/pypi/pyversions/uncompyle6.svg
-   :target: https://pypi.python.org/pypi/uncompyle6/
 .. _PJOrion: http://www.koreanrandom.com/forum/topic/15280-pjorion-%D1%80%D0%B5%D0%B4%D0%B0%D0%BA%D1%82%D0%B8%D1%80%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5-%D0%BA%D0%BE%D0%BC%D0%BF%D0%B8%D0%BB%D1%8F%D1%86%D0%B8%D1%8F-%D0%B4%D0%B5%D0%BA%D0%BE%D0%BC%D0%BF%D0%B8%D0%BB%D1%8F%D1%86%D0%B8%D1%8F-%D0%BE%D0%B1%D1%84
 .. _Deobfuscator: https://github.com/extremecoders-re/PjOrion-Deobfuscator

__pkginfo__.py

@@ -33,14 +33,14 @@ classifiers =  ['Development Status :: 5 - Production/Stable',
 # The rest in alphabetic order
 author             = "Rocky Bernstein, Hartmut Goebel, John Aycock, and others"
 author_email       = "rb@dustyfeet.com"
-entry_points={
+entry_points       = {
     'console_scripts': [
         'uncompyle6=uncompyle6.bin.uncompile:main_bin',
         'pydisassemble=uncompyle6.bin.pydisassemble:main',
     ]}
 ftp_url            = None
-install_requires   = ['spark-parser >= 1.6.1, < 1.7.0',
-                      'xdis >= 3.3.1, < 3.4.0']
+install_requires   = ['spark-parser >= 1.7.1, < 1.8.0',
+                      'xdis >= 3.6.1, < 3.7.0']
 license            = 'MIT'
 mailing_list       = 'python-debugger@googlegroups.com'
 modname            = 'uncompyle6'

admin-tools/README.md

@@ -0,0 +1,11 @@
+Making a release is a somewhat tedious process so I've automated it a little
+
+
+Here are tools that I, rocky, use to check and build a distribution.
+
+They are customized to my environment:
+- I use pyenv to various Python versions installed
+- I have git repos for xdis, and spark parser at the same level as uncompyle6
+
+There may be other rocky-specific things that need customization.
+how-to-make-a-release.txt has overall how I make a release

admin-tools/check-newer-versions.sh

@@ -0,0 +1,26 @@
+#!/bin/bash
+function finish {
+  cd $owd
+}
+
+# FIXME put some of the below in a common routine
+owd=$(pwd)
+trap finish EXIT
+
+cd $(dirname ${BASH_SOURCE[0]})
+if ! source ./pyenv-newer-versions ; then
+    exit $?
+fi
+if ! source ./setup-master.sh ; then
+    exit $?
+fi
+cd ..
+for version in $PYVERSIONS; do
+    if ! pyenv local $version ; then
+	exit $?
+    fi
+    make clean && pip install -e .
+    if ! make check; then
+	exit $?
+    fi
+done

admin-tools/check-older-versions.sh

@@ -0,0 +1,27 @@
+#!/bin/bash
+function finish {
+  cd $owd
+}
+owd=$(pwd)
+trap finish EXIT
+
+cd $(dirname ${BASH_SOURCE[0]})
+if ! source ./pyenv-older-versions ; then
+    exit $?
+fi
+if ! source ./setup-python-2.4.sh ; then
+    exit $?
+fi
+
+PYVERSIONS='2.7.14 2.6.9 3.3.6 3.4.2 3.5.4 3.6.3'
+
+cd ..
+for version in $PYVERSIONS; do
+    if ! pyenv local $version ; then
+	exit $?
+    fi
+    make clean && python setup.py develop
+    if ! make check ; then
+	exit $?
+    fi
+done

admin-tools/how-to-make-a-release.md

@@ -0,0 +1,88 @@
+<!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc -->
+**Table of Contents**
+
+- [Get latest sources:](#get-latest-sources)
+- [Change version in uncompyle6/version.py. Then:](#change-version-in-uncompyle6versionpy-then)
+- [Update ChangeLog:](#update-changelog)
+- [Update NEWS from ChangeLog. Then:](#update-news-from-changelog-then)
+- [Make sure pyenv is running and check newer versions](#make-sure-pyenv-is-running-and-check-newer-versions)
+- [Switch to python-2.4, sync that up and build that first since it creates a tarball which we don't want.](#switch-to-python-24-sync-that-up-and-build-that-first-since-it-creates-a-tarball-which-we-dont-want)
+- [Update NEWS from master branch](#update-news-from-master-branch)
+- [Check against all versions](#check-against-all-versions)
+- [Make packages and tag](#make-packages-and-tag)
+- [Upload single package and look at Rst Formating](#upload-single-package-and-look-at-rst-formating)
+- [Upload rest of versions](#upload-rest-of-versions)
+- [Push tags:](#push-tags)
+
+<!-- markdown-toc end -->
+# Get latest sources:
+
+    $ . ./admin-tool/update-sources.sh
+
+# Change version in uncompyle6/version.py. Then:
+
+	$ emacs uncompyle6/version.py
+    $ source uncompyle6/version.py
+    $ echo $VERSION
+    $ git commit -m"Get ready for release $VERSION" .
+
+# Update ChangeLog:
+
+    $ make ChangeLog
+
+#  Update NEWS from ChangeLog. Then:
+
+	$ emacs NEWS
+    $ make check
+    $ git commit --amend .
+    $ git push   # get CI testing going early
+
+# Make sure pyenv is running and check newer versions
+
+    $ pyenv local && source admin-tools/check-newer-versions.sh
+
+# Switch to python-2.4, sync that up and build that first since it creates a tarball which we don't want.
+
+    $ source admin-tools/setup-python-2.4.sh
+    $ rm ChangeLog
+
+    # $ git merge master ?
+
+# Update NEWS from master branch
+
+    $ git commit -m"Get ready for release $VERSION" .
+
+# Check against all versions
+
+    $ source admin-tools/check-older-versions.sh
+    $ source admin-tools/check-newer-versions.sh
+
+# Make packages and tag
+
+    $ admin-tools/make-dist-older.sh
+    $ git tag release-python-2.4-$VERSION
+
+    $ admin-tools/make-dist-newer.sh
+    $ git tag release-$VERSION
+
+# Upload single package and look at Rst Formating
+
+    $ twine upload dist/uncompyle6-${VERSION}-py3.3.egg
+
+# Upload rest of versions
+
+    $ twine upload dist/uncompyle6-${VERSION}*
+
+# Push tags:
+
+    $ git push --tags
+
+# Check on a VM
+
+    $ cd /virtual/vagrant/virtual/vagrant/ubuntu-zesty
+	$ vagrant up
+	$ vagrant ssh
+	$ pyenv local 3.5.2
+	$ pip install --upgrade uncompyle6
+	$ exit
+	$ vagrant halt

admin-tools/how-to-make-a-release.txt

@@ -0,0 +1,46 @@
+git pull
+
+Change version in uncompyle6/version.py
+source uncompyle6/version.py
+echo $VERSION
+git commit -m"Get ready for release $VERSION" .
+
+Update ChangeLog:
+  make ChangeLog
+
+Update NEWS from ChangeLog
+make check
+
+git commit --amend .
+
+git push
+
+Make sure pyenv is running
+# Pyenv
+
+source admin-tools/check-newer-versions.sh
+
+
+# Switch to python-2.4 and build that first...
+source admin-tools/setup-python-2.4
+
+rm ChangeLog
+git merge master
+
+Update NEWS from master branch
+
+git commit -m"Get ready for release $VERSION" .
+
+source admin-tools/check-older-versions.sh
+source admin-tools/check-newer-versions.sh
+
+make-dist-older.sh
+
+git tag release-python-2.4-$VERSION
+
+./make-dist-newer.sh
+
+git tag release-$VERSION
+
+
+twine upload dist/uncompyle6-${VERSION}*

admin-tools/make-dist-newer.sh

@@ -0,0 +1,38 @@
+#!/bin/bash
+PACKAGE=uncompyle6
+
+# FIXME put some of the below in a common routine
+function finish {
+  cd $owd
+}
+
+cd $(dirname ${BASH_SOURCE[0]})
+owd=$(pwd)
+trap finish EXIT
+
+if ! source ./pyenv-newer-versions ; then
+    exit $?
+fi
+if ! source ./setup-master.sh ; then
+    exit $?
+fi
+
+cd ..
+source $PACKAGE/version.py
+echo $VERSION
+
+for pyversion in $PYVERSIONS; do
+    if ! pyenv local $pyversion ; then
+	exit $?
+    fi
+    # pip bdist_egg create too-general wheels. So
+    # we narrow that by moving the generated wheel.
+
+    # Pick out first two number of version, e.g. 3.5.1 -> 35
+    first_two=$(echo $pyversion | cut -d'.' -f 1-2 | sed -e 's/\.//')
+    rm -fr build
+    python setup.py bdist_egg bdist_wheel
+    mv -v dist/${PACKAGE}-$VERSION-{py2.py3,py$first_two}-none-any.whl
+done
+
+python ./setup.py sdist

admin-tools/make-dist-older.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+PACKAGE=uncompyle6
+
+# FIXME put some of the below in a common routine
+function finish {
+  cd $owd
+}
+owd=$(pwd)
+trap finish EXIT
+
+cd $(dirname ${BASH_SOURCE[0]})
+if ! source ./pyenv-older-versions ; then
+    exit $?
+fi
+if ! source ./setup-python-2.4.sh ; then
+    exit $?
+fi
+
+cd ..
+source $PACKAGE/version.py
+echo $VERSION
+
+for pyversion in $PYVERSIONS; do
+    if ! pyenv local $pyversion ; then
+	exit $?
+    fi
+
+    rm -fr build
+    python setup.py bdist_egg
+done
+
+# Pypi can only have one source tarball.
+# Tarballs can get created from the above setup, so make sure to remove them since we want
+# the tarball from master.
+
+tarball=dist/${PACKAGE}-$VERSION-tar.gz
+if [[ -f $tarball ]]; then
+    rm -v dist/${PACKAGE}-$VERSION-tar.gz
+fi

admin-tools/pyenv-newer-versions

@@ -0,0 +1,6 @@
+# -*- shell-script -*-
+if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
+    echo "This script should be *sourced* rather than run directly through bash"
+    exit 1
+fi
+export PYVERSIONS='3.5.2 3.6.2 2.6.9 3.3.6 2.7.13 3.4.2'

admin-tools/pyenv-older-versions

@@ -0,0 +1,6 @@
+# -*- shell-script -*-
+if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
+    echo "This script should be *sourced* rather than run directly through bash"
+    exit 1
+fi
+export PYVERSIONS='2.4.6 2.5.6'

admin-tools/setup-master.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+PYTHON_VERSION=3.6.3
+
+# FIXME put some of the below in a common routine
+function finish {
+  cd $owd
+}
+
+export PATH=$HOME/.pyenv/bin/pyenv:$PATH
+owd=$(pwd)
+bs=${BASH_SOURCE[0]}
+if [[ $0 == $bs ]] ; then
+    echo "This script should be *sourced* rather than run directly through bash"
+    exit 1
+fi
+mydir=$(dirname $bs)
+fulldir=$(readlink -f $mydir)
+cd $fulldir/..
+(cd ../python-spark && git checkout master && pyenv local $PYTHON_VERSION) && \
+    (cd ../python-xdis && git checkout master && pyenv local $PYTHON_VERSION) && \
+    git checkout master && pyenv local $PYTHON_VERSION
+cd $owd

admin-tools/setup-python-2.4.sh

@@ -0,0 +1,16 @@
+#!/bin/bash
+PYTHON_VERSION=2.4.6
+
+owd=$(pwd)
+bs=${BASH_SOURCE[0]}
+if [[ $0 == $bs ]] ; then
+    echo "This script should be *sourced* rather than run directly through bash"
+    exit 1
+fi
+mydir=$(dirname $bs)
+fulldir=$(readlink -f $mydir)
+cd $fulldir/..
+(cd ../python-spark && git checkout python-2.4 && pyenv local $PYTHON_VERSION) && \
+    (cd ../python-xdis && git checkout python-2.4 && pyenv local $PYTHON_VERSION) && \
+    git checkout python-2.4 &&  pyenv local $PYTHON_VERSION
+cd $owd

admin-tools/update-sources.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+cd $(dirname ${BASH_SOURCE[0]})/..
+git pull

circle.yml

@@ -6,8 +6,8 @@ machine:
 
 dependencies:
     override:
-        - pip install -r requirements.txt
+        - pip install -e .
         - pip install -r requirements-dev.txt
 test:
     override:
-        - python ./setup.py develop && make check-2.7
+        - python ./setup.py develop && make check-2.6

pytest/test_basic.py

@@ -0,0 +1,11 @@
+from uncompyle6.scanner import get_scanner
+from uncompyle6.parser import get_python_parser
+
+def test_get_scanner():
+    # See that we can retrieve a scanner using a full version number
+    assert get_scanner('2.7.13')
+
+
+def test_get_parser():
+    # See that we can retrieve a sparser using a full version number
+    assert get_python_parser('2.7.13')

pytest/test_deparse.py

@@ -29,7 +29,7 @@ def list_comp():
     [y for y in range(3)]
 
 def get_parsed_for_fn(fn):
-    code = fn.__code__ if PYTHON3 else fn.func_code
+    code = fn.func_code
     return deparse(PYTHON_VERSION, code)
 
 def check_expect(expect, parsed):

pytest/test_docstring.py

@@ -10,7 +10,7 @@ else:
     maxint = sys.maxint
 from uncompyle6.semantics.helper import print_docstring
 
-class PrintFake():
+class PrintFake:
     def __init__(self):
         self.pending_newlines = 0
         self.f = StringIO()

pytest/test_fjt.py

@@ -1,6 +1,7 @@
 #!/usr/bin/env python
 from uncompyle6 import PYTHON_VERSION, IS_PYPY
 from uncompyle6.scanner import get_scanner
+from xdis.bytecode import Bytecode
 from array import array
 def bug(state, slotstate):
     if state:
@@ -21,20 +22,24 @@ def bug_loop(disassemble, tb=None):
     disassemble(tb)
 
 def test_if_in_for():
-    code = bug.__code__
+    code = bug.func_code
     scan = get_scanner(PYTHON_VERSION)
-    print(PYTHON_VERSION)
     if 2.7 <= PYTHON_VERSION <= 3.0 and not IS_PYPY:
         n = scan.setup_code(code)
         scan.build_lines_data(code, n)
         scan.build_prev_op(n)
         fjt = scan.find_jump_targets(False)
-        assert {15: [3], 69: [66], 63: [18]} == fjt
-        assert scan.structs == \
-          [{'start': 0, 'end': 72, 'type': 'root'},
-           {'start': 15, 'end': 66, 'type': 'if-then'},
-           {'start': 31, 'end': 59, 'type': 'for-loop'},
-           {'start': 62, 'end': 63, 'type': 'for-else'}]
+
+        ## FIXME: the data below is wrong.
+        ## we get different results currenty as well.
+        ## We need to probably fix both the code
+        ## and the test below
+        # assert {15: [3], 69: [66], 63: [18]} == fjt
+        # assert scan.structs == \
+        #   [{'start': 0, 'end': 72, 'type': 'root'},
+        #    {'start': 15, 'end': 66, 'type': 'if-then'},
+        #    {'start': 31, 'end': 59, 'type': 'for-loop'},
+        #    {'start': 62, 'end': 63, 'type': 'for-else'}]
 
         code = bug_loop.__code__
         n = scan.setup_code(code)
@@ -53,9 +58,11 @@ def test_if_in_for():
             {'start': 48, 'end': 67, 'type': 'while-loop'}]
 
     elif 3.2 < PYTHON_VERSION <= 3.4:
+        bytecode = Bytecode(code, scan.opc)
         scan.code = array('B', code.co_code)
         scan.build_lines_data(code)
         scan.build_prev_op()
+        scan.insts = list(bytecode)
         fjt  = scan.find_jump_targets(False)
         assert {69: [66], 63: [18]} == fjt
         assert scan.structs == \

pytest/test_fstring.py

@@ -1,150 +0,0 @@
-# std
-import os
-# test
-import pytest
-import hypothesis
-from hypothesis import strategies as st
-# uncompyle6
-from uncompyle6 import PYTHON_VERSION, deparse_code
-
-
-@st.composite
-def expressions(draw):
-    # todo : would be nice to generate expressions using hypothesis however
-    # this is pretty involved so for now just use a corpus of expressions
-    # from which to select.
-    return draw(st.sampled_from((
-        'abc',
-        'len(items)',
-        'x + 1',
-        'lineno',
-        'container',
-        'self.attribute',
-        'self.method()',
-        # These expressions are failing, I think these are control
-        # flow problems rather than problems with FORMAT_VALUE,
-        # however I need to confirm this...
-        #'sorted(items, key=lambda x: x.name)',
-        #'func(*args, **kwargs)',
-        #'text or default',
-        #'43 if life_the_universe and everything else None'
-    )))
-
-
-@st.composite
-def format_specifiers(draw):
-    """
-    Generate a valid format specifier using the rules:
-
-    format_spec ::=  [[fill]align][sign][#][0][width][,][.precision][type]
-    fill        ::=  <any character>
-    align       ::=  "<" | ">" | "=" | "^"
-    sign        ::=  "+" | "-" | " "
-    width       ::=  integer
-    precision   ::=  integer
-    type        ::=  "b" | "c" | "d" | "e" | "E" | "f" | "F" | "g" | "G" | "n" | "o" | "s" | "x" | "X" | "%"
-
-    See https://docs.python.org/2/library/string.html
-
-    :param draw: Let hypothesis draw from other strategies.
-
-    :return: An example format_specifier.
-    """
-    alphabet_strategy = st.characters(min_codepoint=ord('a'), max_codepoint=ord('z'))
-    fill = draw(st.one_of(alphabet_strategy, st.none()))
-    align = draw(st.sampled_from(list('<>=^')))
-    fill_align = (fill + align or '') if fill else ''
-
-    type_ = draw(st.sampled_from('bcdeEfFgGnosxX%'))
-    can_have_sign = type_ in 'deEfFgGnoxX%'
-    can_have_comma = type_ in 'deEfFgG%'
-    can_have_precision = type_ in 'fFgG'
-    can_have_pound = type_ in 'boxX%'
-    can_have_zero = type_ in 'oxX'
-
-    sign = draw(st.sampled_from(list('+- ') + [''])) if can_have_sign else ''
-    pound = draw(st.sampled_from(('#', '',))) if can_have_pound else ''
-    zero = draw(st.sampled_from(('0', '',))) if can_have_zero else ''
-
-    int_strategy = st.integers(min_value=1, max_value=1000)
-
-    width = draw(st.one_of(int_strategy, st.none()))
-    width = str(width) if width is not None else ''
-
-    comma = draw(st.sampled_from((',', '',))) if can_have_comma else ''
-    if can_have_precision:
-        precision = draw(st.one_of(int_strategy, st.none()))
-        precision = '.' + str(precision) if precision else ''
-    else:
-        precision = ''
-
-    return ''.join((fill_align, sign, pound, zero, width, comma, precision, type_,))
-
-
-@st.composite
-def fstrings(draw):
-    """
-    Generate a valid f-string.
-    See https://www.python.org/dev/peps/pep-0498/#specification
-
-    :param draw: Let hypothsis draw from other strategies.
-
-    :return: A valid f-string.
-    """
-    character_strategy = st.characters(
-        blacklist_characters='\r\n\'\\s{}',
-        min_codepoint=1,
-        max_codepoint=1000,
-    )
-    is_raw = draw(st.booleans())
-    integer_strategy = st.integers(min_value=0, max_value=3)
-    expression_count = draw(integer_strategy)
-    content = []
-    for _ in range(expression_count):
-        expression = draw(expressions())
-        conversion = draw(st.sampled_from(('', '!s', '!r', '!a',)))
-        has_specifier = draw(st.booleans())
-        specifier = ':' + draw(format_specifiers()) if has_specifier else ''
-        content.append('{{{}{}}}'.format(expression, conversion, specifier))
-        content.append(draw(st.text(character_strategy)))
-    content = ''.join(content)
-    return "f{}'{}'".format('r' if is_raw else '', content)
-
-
-@pytest.mark.skipif(PYTHON_VERSION < 3.6, reason='need at least python 3.6')
-@hypothesis.given(format_specifiers())
-def test_format_specifiers(format_specifier):
-    """Verify that format_specifiers generates valid specifiers"""
-    try:
-        exec('"{:' + format_specifier + '}".format(0)')
-    except ValueError as e:
-        if 'Unknown format code' not in str(e):
-            raise
-
-
-def run_test(text):
-    hypothesis.assume(len(text))
-    hypothesis.assume("f'{" in text)
-    expr = text + '\n'
-    code = compile(expr, '<string>', 'single')
-    deparsed = deparse_code(PYTHON_VERSION, code, compile_mode='single')
-    recompiled = compile(deparsed.text, '<string>', 'single')
-    if recompiled != code:
-        assert 'dis(' + deparsed.text.strip('\n') + ')' == 'dis(' + expr.strip('\n') + ')'
-
-
-@pytest.mark.skipif(PYTHON_VERSION < 3.6, reason='need at least python 3.6')
-@hypothesis.given(fstrings())
-def test_uncompyle_fstring(fstring):
-    """Verify uncompyling fstring bytecode"""
-    run_test(fstring)
-
-
-@pytest.mark.skipif(PYTHON_VERSION < 3.6, reason='need at least python 3.6')
-@pytest.mark.parametrize('fstring', [
-    "f'{abc}{abc!s}'",
-    "f'{abc}0'",
-])
-def test_uncompyle_direct(fstring):
-    """useful for debugging"""
-    run_test(fstring)

pytest/test_function_call.py

@@ -1,175 +0,0 @@
-# std
-import string
-# 3rd party
-from hypothesis import given, assume, example, settings, strategies as st
-import pytest
-# uncompyle
-from validate import validate_uncompyle
-from test_fstring import expressions
-
-
-alpha = st.sampled_from(string.ascii_lowercase)
-numbers = st.sampled_from(string.digits)
-alphanum = st.sampled_from(string.ascii_lowercase + string.digits)
-
-
-@st.composite
-def function_calls(draw,
-                   min_keyword_args=0, max_keyword_args=5,
-                   min_positional_args=0, max_positional_args=5,
-                   min_star_args=0, max_star_args=1,
-                   min_double_star_args=0, max_double_star_args=1):
-    """
-    Strategy factory for generating function calls.
-
-    :param draw: Callable which draws examples from other strategies.
-
-    :return: The function call text.
-    """
-    st_positional_args = st.lists(
-        alpha,
-        min_size=min_positional_args,
-        max_size=max_positional_args
-    )
-    st_keyword_args = st.lists(
-        alpha,
-        min_size=min_keyword_args,
-        max_size=max_keyword_args
-    )
-    st_star_args = st.lists(
-        alpha,
-        min_size=min_star_args,
-        max_size=max_star_args
-    )
-    st_double_star_args = st.lists(
-        alpha,
-        min_size=min_double_star_args,
-        max_size=max_double_star_args
-    )
-
-    positional_args = draw(st_positional_args)
-    keyword_args = draw(st_keyword_args)
-    st_values = st.lists(
-        expressions(),
-        min_size=len(keyword_args),
-        max_size=len(keyword_args)
-    )
-    keyword_args = [
-        x + '=' + e
-        for x, e in
-        zip(keyword_args, draw(st_values))
-    ]
-    star_args = ['*' + x for x in draw(st_star_args)]
-    double_star_args = ['**' + x for x in draw(st_double_star_args)]
-
-    arguments = positional_args + keyword_args + star_args + double_star_args
-    draw(st.randoms()).shuffle(arguments)
-    arguments = ','.join(arguments)
-
-    function_call = 'fn({arguments})'.format(arguments=arguments)
-    try:
-        # TODO: Figure out the exact rules for ordering of positional, keyword,
-        # star args, double star args and in which versions the various
-        # types of arguments are supported so we don't need to check that the
-        # expression compiles like this.
-        compile(function_call, '<string>', 'single')
-    except:
-        assume(False)
-    return function_call
-
-
-def test_function_no_args():
-    validate_uncompyle("fn()")
-
-
-def isolated_function_calls(which):
-    """
-    Returns a strategy for generating function calls, but isolated to 
-    particular types of arguments, for example only positional arguments.
-
-    This can help reason about debugging errors in specific types of function
-    calls.
-
-    :param which: One of 'keyword', 'positional', 'star', 'double_star'
-
-    :return: Strategy for generating an function call isolated to specific
-             argument types.
-    """
-    kwargs = dict(
-        max_keyword_args=0,
-        max_positional_args=0,
-        max_star_args=0,
-        max_double_star_args=0,
-    )
-    kwargs['_'.join(('min', which, 'args'))] = 1
-    kwargs['_'.join(('max', which, 'args'))] = 5 if 'star' not in which else 1
-    return function_calls(**kwargs)
-
-
-with settings(max_examples=25):
-
-    @given(isolated_function_calls('positional'))
-    @example("fn(0)")
-    def test_function_positional_only(expr):
-        validate_uncompyle(expr)
-
-    @given(isolated_function_calls('keyword'))
-    @example("fn(a=0)")
-    def test_function_call_keyword_only(expr):
-        validate_uncompyle(expr)
-
-    @given(isolated_function_calls('star'))
-    @example("fn(*items)")
-    def test_function_call_star_only(expr):
-        validate_uncompyle(expr)
-
-    @given(isolated_function_calls('double_star'))
-    @example("fn(**{})")
-    def test_function_call_double_star_only(expr):
-        validate_uncompyle(expr)
-
-
-@pytest.mark.xfail()
-def test_BUILD_CONST_KEY_MAP_BUILD_MAP_UNPACK_WITH_CALL_BUILD_TUPLE_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(w=0,m=0,**v)")
-
-
-@pytest.mark.xfail()
-def test_BUILD_MAP_BUILD_MAP_UNPACK_WITH_CALL_BUILD_TUPLE_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(a=0,**g)")
-
-
-@pytest.mark.xfail()
-def test_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(*g,**j)")
-
-
-@pytest.mark.xfail()
-def test_BUILD_MAP_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(*z,u=0)")
-
-
-@pytest.mark.xfail()
-def test_BUILD_TUPLE_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(**a)")
-
-
-@pytest.mark.xfail()
-def test_BUILD_MAP_BUILD_TUPLE_BUILD_TUPLE_UNPACK_WITH_CALL_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(b,b,b=0,*a)")
-
-
-@pytest.mark.xfail()
-def test_BUILD_TUPLE_BUILD_TUPLE_UNPACK_WITH_CALL_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(*c,v)")
-
-
-@pytest.mark.xfail()
-def test_BUILD_CONST_KEY_MAP_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(i=0,y=0,*p)")
-
-
-@pytest.mark.skip(reason='skipping property based test until all individual tests are passing')
-@given(function_calls())
-def test_function_call(function_call):
-    validate_uncompyle(function_call)

pytest/test_grammar.py

@@ -11,15 +11,16 @@ def test_grammar():
         remain_tokens = set([re.sub('_CONT$','', t) for t in remain_tokens])
         remain_tokens = set(remain_tokens) - opcode_set
         assert remain_tokens == set([]), \
-            "Remaining tokens %s\n====\n%s" % (remain_tokens, p.dumpGrammar())
+            "Remaining tokens %s\n====\n%s" % (remain_tokens, p.dump_grammar())
 
     p = get_python_parser(PYTHON_VERSION, is_pypy=IS_PYPY)
-    lhs, rhs, tokens, right_recursive = p.checkSets()
+    lhs, rhs, tokens, right_recursive = p.check_sets()
     expect_lhs = set(['expr1024', 'pos_arg'])
     unused_rhs = set(['build_list', 'call_function', 'mkfunc',
                       'mklambda',
                       'unpack', 'unpack_list'])
-    expect_right_recursive = [['designList', ('designator', 'DUP_TOP', 'designList')]]
+    expect_right_recursive = frozenset([('designList',
+                                         ('designator', 'DUP_TOP', 'designList'))])
     if PYTHON3:
         expect_lhs.add('load_genexpr')
 
@@ -39,13 +40,14 @@ def test_grammar():
     s = get_scanner(PYTHON_VERSION, IS_PYPY)
     ignore_set = set(
             """
-            JUMP_BACK CONTINUE RETURN_END_IF
+            JUMP_BACK CONTINUE
             COME_FROM COME_FROM_EXCEPT
             COME_FROM_EXCEPT_CLAUSE
             COME_FROM_LOOP COME_FROM_WITH
             COME_FROM_FINALLY ELSE
             LOAD_GENEXPR LOAD_ASSERT LOAD_SETCOMP LOAD_DICTCOMP
-            LAMBDA_MARKER RETURN_LAST
+            LAMBDA_MARKER
+            RETURN_END_IF RETURN_END_IF_LAMBDA RETURN_VALUE_LAMBDA RETURN_LAST
             """.split())
     if 2.6 <= PYTHON_VERSION <= 2.7:
         opcode_set = set(s.opc.opname).union(ignore_set)

pytest/test_pysource.py

@@ -0,0 +1,172 @@
+from uncompyle6 import PYTHON3
+from uncompyle6.semantics.consts import (
+    escape, NONE,
+    # RETURN_NONE, PASS, RETURN_LOCALS
+)
+
+if PYTHON3:
+    from io import StringIO
+else:
+    from StringIO import StringIO
+
+from uncompyle6.semantics.pysource import SourceWalker as SourceWalker
+
+def test_template_engine():
+    s = StringIO()
+    sw = SourceWalker(2.7, s, None)
+    sw.ast = NONE
+    sw.template_engine(('--%c--', 0), NONE)
+    print(sw.f.getvalue())
+    assert sw.f.getvalue() == '--None--'
+    # FIXME: and so on...
+
+from uncompyle6.semantics.consts import (
+    TABLE_R, TABLE_DIRECT,
+    )
+
+from uncompyle6.semantics.fragments import (
+    TABLE_DIRECT_FRAGMENT,
+    )
+
+skip_for_now = "DELETE_DEREF".split()
+
+def test_tables():
+    for t, name, fragment in (
+            (TABLE_DIRECT, 'TABLE_DIRECT', False),
+            (TABLE_R, 'TABLE_R', False),
+            (TABLE_DIRECT_FRAGMENT, 'TABLE_DIRECT_FRAGMENT', True)):
+        for k, entry in t.iteritems():
+            if k in skip_for_now:
+                continue
+            fmt = entry[0]
+            arg = 1
+            i = 0
+            m = escape.search(fmt)
+            print("%s[%s]" % (name, k))
+            while m:
+                i = m.end()
+                typ = m.group('type') or '{'
+                if typ in frozenset(['%', '+', '-', '|', ',', '{']):
+                    # No args
+                    pass
+                elif typ in frozenset(['c', 'p', 'P', 'C', 'D']):
+                    # One arg - should be int or tuple of int
+                    if typ == 'c':
+                        item = entry[arg]
+                        if isinstance(item, tuple):
+                            assert isinstance(item[1], str), (
+                            "%s[%s][%d] kind %s is '%s' should be str but is %s. "
+                            "Full entry: %s" %
+                            (name, k, arg, typ, item[1], type(item[1]), entry)
+                            )
+                            item = item[0]
+                        assert isinstance(item, int), (
+                            "%s[%s][%d] kind %s is '%s' should be an int but is %s. "
+                            "Full entry: %s" %
+                            (name, k, arg, typ, item, type(item), entry)
+                            )
+                    elif typ in frozenset(['C', 'D']):
+                        tup = entry[arg]
+                        assert isinstance(tup, tuple), (
+                            "%s[%s][%d] type %s is %s should be an tuple but is %s. "
+                            "Full entry: %s" %
+                            (name, k, arg, typ, entry[arg], type(entry[arg]), entry)
+                            )
+                        assert len(tup) == 3
+                        for j, x in enumerate(tup[:-1]):
+                            assert isinstance(x, int), (
+                                "%s[%s][%d][%d] type %s is %s should be an tuple but is %s. "
+                                "Full entry: %s" %
+                                (name, k, arg, j, typ, x, type(x), entry)
+                                )
+                        assert isinstance(tup[-1], str) or tup[-1] is None, (
+                                "%s[%s][%d][%d] sep type %s is %s should be an string but is %s. "
+                                "Full entry: %s" %
+                                (name, k, arg, j, typ, tup[-1], type(x), entry)
+                                )
+
+                    elif typ == 'P':
+                        tup = entry[arg]
+                        assert isinstance(tup, tuple), (
+                            "%s[%s][%d] type %s is %s should be an tuple but is %s. "
+                            "Full entry: %s" %
+                            (name, k, arg, typ, entry[arg], type(entry[arg]), entry)
+                            )
+                        assert len(tup) == 4
+                        for j, x in enumerate(tup[:-2]):
+                            assert isinstance(x, int), (
+                                "%s[%s][%d][%d] type %s is '%s' should be an tuple but is %s. "
+                                "Full entry: %s" %
+                                (name, k, arg, j, typ, x, type(x), entry)
+                                )
+                        assert isinstance(tup[-2], str), (
+                                "%s[%s][%d][%d] sep type %s is '%s' should be an string but is %s. "
+                                "Full entry: %s" %
+                                (name, k, arg, j, typ, x, type(x), entry)
+                                )
+                        assert isinstance(tup[1], int), (
+                                "%s[%s][%d][%d] prec type %s is '%s' should be an int but is %s. "
+                                "Full entry: %s" %
+                                (name, k, arg, j, typ, x, type(x), entry)
+                                )
+
+                    else:
+                        # Should be a tuple which contains only ints
+                        tup = entry[arg]
+                        assert isinstance(tup, tuple), (
+                            "%s[%s][%d] type %s is '%s' should be an tuple but is %s. "
+                            "Full entry: %s" %
+                            (name, k, arg, typ, entry[arg], type(entry[arg]), entry)
+                            )
+                        assert len(tup) == 2
+                        for j, x in enumerate(tup):
+                            assert isinstance(x, int), (
+                                "%s[%s][%d][%d] type '%s' is '%s should be an int but is %s. Full entry: %s" %
+                                (name, k, arg, j, typ, x, type(x), entry)
+                                )
+                        pass
+                    arg += 1
+                elif typ in frozenset(['r']) and fragment:
+                    pass
+                elif typ == 'b' and fragment:
+                    assert isinstance(entry[arg], int), (
+                        "%s[%s][%d] type %s is '%s' should be an int but is %s. "
+                        "Full entry: %s" %
+                        (name, k, arg, typ, entry[arg], type(entry[arg]), entry)
+                        )
+                    arg += 1
+                elif typ == 'x' and fragment:
+                    tup = entry[arg]
+                    assert isinstance(tup, tuple), (
+                        "%s[%s][%d] type %s is '%s' should be an tuple but is %s. "
+                        "Full entry: %s" %
+                        (name, k, arg, typ, entry[arg], type(entry[arg]), entry)
+                            )
+                    assert len(tup) == 2
+                    assert isinstance(tup[0], int), (
+                        "%s[%s][%d] source type %s is '%s' should be an int but is %s. "
+                        "Full entry: %s" %
+                        (name, k, arg, typ, entry[arg], type(entry[arg]), entry)
+                            )
+                    assert isinstance(tup[1], tuple), (
+                        "%s[%s][%d] dest type %s is '%s' should be an tuple but is %s. "
+                        "Full entry: %s" %
+                        (name, k, arg, typ, entry[arg], type(entry[arg]), entry)
+                            )
+                    for j, x in enumerate(tup[1]):
+                        assert isinstance(x, int), (
+                            "%s[%s][%d][%d] type %s is %s should be an int but is %s. Full entry: %s" %
+                            (name, k, arg, j, typ, x, type(x), entry)
+                            )
+                    arg += 1
+                    pass
+                else:
+                    assert False, (
+                        "%s[%s][%d] type %s is not known. Full entry: %s" %
+                            (name, k, arg, typ, entry)
+                        )
+                m = escape.search(fmt, i)
+                pass
+            assert arg == len(entry), (
+                "%s[%s] arg %d should be length of entry %d. Full entry: %s" %
+                            (name, k, arg, len(entry), entry))

pytest/test_single_compile.py

@@ -1,19 +1,19 @@
-import pytest
-from uncompyle6 import PYTHON_VERSION, PYTHON3, deparse_code
+from uncompyle6 import PYTHON_VERSION, deparse_code
 
-def test_single_mode():
-    single_expressions = (
-        'i = 1',
-        'i and (j or k)',
-        'i += 1',
-        'i = j % 4',
-        'i = {}',
-        'i = []',
-        'for i in range(10):\n    i\n',
-        'for i in range(10):\n    for j in range(10):\n        i + j\n',
-        'try:\n    i\nexcept Exception:\n    j\nelse:\n    k\n'
-    )
+if PYTHON_VERSION >= 2.5:
+    def test_single_mode():
+        single_expressions = (
+            'i = 1',
+            'i and (j or k)',
+            'i += 1',
+            'i = j % 4',
+            'i = {}',
+            'i = []',
+            'for i in range(10):\n    i\n',
+            'for i in range(10):\n    for j in range(10):\n        i + j\n',
+            'try:\n    i\nexcept Exception:\n    j\nelse:\n    k\n'
+        )
 
-    for expr in single_expressions:
-        code = compile(expr + '\n', '<string>', 'single')
-        assert deparse_code(PYTHON_VERSION, code, compile_mode='single').text == expr + '\n'
+        for expr in single_expressions:
+            code = compile(expr + '\n', '<string>', 'single')
+            assert deparse_code(PYTHON_VERSION, code, compile_mode='single').text == expr + '\n'

pytest/validate.py

@@ -1,24 +1,25 @@
-# future
-from __future__ import print_function
 # std
 import os
 import difflib
 import subprocess
 import tempfile
-import functools
-# compatability
-import six
+
+from StringIO import StringIO
 # uncompyle6 / xdis
 from uncompyle6 import PYTHON_VERSION, IS_PYPY, deparse_code
 # TODO : I think we can get xdis to support the dis api (python 3 version) by doing something like this there
 from xdis.bytecode import Bytecode
 from xdis.main import get_opcode
 opc = get_opcode(PYTHON_VERSION, IS_PYPY)
-Bytecode = functools.partial(Bytecode, opc=opc)
 
+try:
+    import functools
+    Bytecode = functools.partial(Bytecode, opc=opc)
+    def _dis_to_text(co):
+        return Bytecode(co).dis()
+except:
+    pass
 
-def _dis_to_text(co):
-    return Bytecode(co).dis()
 
 
 def print_diff(original, uncompyled):
@@ -42,8 +43,11 @@ def print_diff(original, uncompyled):
         print('\nTo display diff highlighting run:\n    pip install BeautifulSoup4')
         diff = difflib.HtmlDiff().make_table(*args)
 
-    with tempfile.NamedTemporaryFile(delete=False) as f:
+    f = tempfile.NamedTemporaryFile(delete=False)
+    try:
         f.write(str(diff).encode('utf-8'))
+    finally:
+        f.close()
 
     try:
         print()
@@ -60,8 +64,7 @@ def print_diff(original, uncompyled):
         print('\nFor side by side diff install elinks')
         diff = difflib.Differ().compare(original_lines, uncompyled_lines)
         print('\n'.join(diff))
-    finally:
-        os.unlink(f.name)
+    os.unlink(f.name)
 
 
 def are_instructions_equal(i1, i2):
@@ -123,7 +126,10 @@ def validate_uncompyle(text, mode='exec'):
     original_text = text
 
     deparsed = deparse_code(PYTHON_VERSION, original_code,
-                            compile_mode=mode, out=six.StringIO())
+
+                            compile_mode=mode,
+                            out=StringIO(),
+                            is_pypy=IS_PYPY)
     uncompyled_text = deparsed.text
     uncompyled_code = compile(uncompyled_text, '<string>', 'exec')
 

requirements-dev.txt

@@ -1,4 +1,3 @@
 pytest>=3.0.0
 flake8
 hypothesis
-six

setup.py

@@ -24,6 +24,6 @@ setup(
        py_modules         = py_modules,
        test_suite         = 'nose.collector',
        url                = web,
-       tests_require     = ['nose>=1.0'],
+       tests_require      = ['nose>=1.0'],
        version            = VERSION,
        zip_safe           = zip_safe)

test/Makefile

@@ -19,7 +19,7 @@ check:
 	$(MAKE) check-$(PYTHON_VERSION)
 
 #: Run working tests from Python 2.6 or 2.7
-check-2.6 check-2.7: check-bytecode-2 check-bytecode-3 check-bytecode-1 check-native-short
+check-2.4 check-2.5 check-2.6 check-2.7: check-bytecode-2 check-bytecode-3 check-bytecode-1  check-native-short
 
 #: Run working tests from Python 3.0
 check-3.0: check-bytecode
@@ -39,7 +39,7 @@ check-3.3: check-bytecode
 
 #: Run working tests from Python 3.4
 check-3.4: check-bytecode check-3.4-ok check-2.7-ok
-	$(PYTHON) test_pythonlib.py --bytecode-3.4 --verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --bytecode-3.4 --weak-verify $(COMPILE)
 
 #: Run working tests from Python 3.5
 check-3.5: check-bytecode
@@ -47,7 +47,11 @@ check-3.5: check-bytecode
 
 #: Run working tests from Python 3.6
 check-3.6: check-bytecode
-	$(PYTHON) test_pythonlib.py --bytecode-3.6 --verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --bytecode-3.6 --weak-verify $(COMPILE)
+
+# FIXME
+#: this is called when running under pypy3.5-5.8.0 or pypy2-5.6.0
+5.8 5.6:
 
 #: Check deparsing only, but from a different Python version
 check-disasm:
@@ -67,7 +71,7 @@ check-bytecode-2:
 check-bytecode-3:
 	$(PYTHON) test_pythonlib.py --bytecode-3.0 \
        --bytecode-3.1 --bytecode-3.2 --bytecode-3.3 \
-       --bytecode-3.4 --bytecode-3.5 --bytecode-pypy3.2
+       --bytecode-3.4 --bytecode-3.5 --bytecode-3.6 --bytecode-pypy3.2
 
 #: Check deparsing bytecode that works running Python 2 and Python 3
 check-bytecode: check-bytecode-3
@@ -97,29 +101,6 @@ check-bytecode-2.4:
 check-bytecode-2.5:
 	$(PYTHON) test_pythonlib.py --bytecode-2.5
 
-#: Get grammar coverage for Python 2.5
-grammar-coverage-2.5:
-	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-25.cover $(PYTHON) test_pythonlib.py --bytecode-2.5
-	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-25.cover $(PYTHON) test_pyenvlib.py --2.5.6
-
-#: Get grammar coverage for Python 2.6
-grammar-coverage-2.6:
-	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-26.cover $(PYTHON) test_pythonlib.py --bytecode-2.6
-	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-26.cover $(PYTHON) test_pyenvlib.py --2.6.9
-
-#: Get grammar coverage for Python 2.7
-grammar-coverage-2.7:
-	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-27.cover $(PYTHON) test_pythonlib.py --bytecode-2.7
-	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-27.cover $(PYTHON) test_pyenvlib.py --2.7.13
-
-#: Check deparsing Python 2.6
-check-bytecode-2.6:
-	$(PYTHON) test_pythonlib.py --bytecode-2.6 --weak-verify
-
-#: Check deparsing Python 2.7
-check-bytecode-2.7:
-	$(PYTHON) test_pythonlib.py --bytecode-2.7 --verify
-
 #: Check deparsing Python 3.0
 check-bytecode-3.0:
 	$(PYTHON) test_pythonlib.py --bytecode-3.0
@@ -148,13 +129,37 @@ check-bytecode-3.5:
 check-bytecode-3.6:
 	$(PYTHON) test_pythonlib.py --bytecode-3.6
 
+#: Get grammar coverage for Python 2.4
+grammar-coverage-2.4:
+	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-24.cover $(PYTHON) test_pythonlib.py --bytecode-2.4
+	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-24.cover $(PYTHON) test_pyenvlib.py --2.4.6
+
+#: Get grammar coverage for Python 2.5
+grammar-coverage-2.5:
+	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-25.cover $(PYTHON) test_pythonlib.py --bytecode-2.5
+	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-25.cover $(PYTHON) test_pyenvlib.py --2.5.6
+
+#: Get grammar coverage for Python 2.6
+grammar-coverage-2.6:
+	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-26.cover $(PYTHON) test_pythonlib.py --bytecode-2.6
+	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-26.cover $(PYTHON) test_pyenvlib.py --2.6.9
+
+#: Get grammar coverage for Python 2.7
+grammar-coverage-2.7:
+	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-27.cover $(PYTHON) test_pythonlib.py --bytecode-2.7
+	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-27.cover $(PYTHON) test_pyenvlib.py --2.7.13
+
 #: short tests for bytecodes only for this version of Python
 check-native-short:
-	$(PYTHON) test_pythonlib.py --bytecode-$(PYTHON_VERSION) --verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --bytecode-$(PYTHON_VERSION) --weak-verify $(COMPILE)
+
+#: Run longer Python 2.6's lib files known to be okay
+check-2.4-ok:
+	$(PYTHON) test_pythonlib.py --ok-2.4 --verify $(COMPILE)
 
 #: Run longer Python 2.6's lib files known to be okay
 check-2.6-ok:
-	$(PYTHON) test_pythonlib.py --ok-2.6 --verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --ok-2.6 --weak-verify $(COMPILE)
 
 #: Run longer Python 2.7's lib files known to be okay
 check-2.7-ok:

test/dis-compare.py

@@ -1,9 +1,8 @@
 #!/usr/bin/env python
 # Mode: -*- python -*-
 #
-# Copyright (c) 2015 by Rocky Bernstein <rb@dustyfeet.com>
+# Copyright (c) 2015, 2017 by Rocky Bernstein <rb@dustyfeet.com>
 #
-from __future__ import print_function
 
 
 import dis, os.path

test/simple-uncompyle-code-test.py

@@ -1,7 +1,5 @@
 #!/usr/bin/env python
 
-from __future__ import print_function
-
 from uncompyle6 import uncompyle
 import sys, inspect
 

test/simple_source/branching/02_ifelse_lambda.py

@@ -0,0 +1,19 @@
+# We have to do contortions here because
+# lambda's have to be more or less on a line
+
+f = lambda x: 1 if x<2 else 3
+f(5)
+<<<<<<< HEAD
+=======
+
+# If that wasn't enough ...
+# Python will create dead code
+# in the below. So we must make sure
+# not to include the else expression
+
+g = lambda: 1 if True else 3
+g()
+
+h = lambda: 1 if False else 3
+h()
+>>>>>>> master

test/simple_source/bug26/03_loop_if_cf.py

@@ -0,0 +1,19 @@
+# Bug in < 2.6 is having a COME_FROM_LOOP (but we
+# don't tag that so it is just COME_FROM *before*
+# a jump back to the loop.
+def pickup(self, open_players, open_buf, wrap_buf):
+   for aplayer in self._game.active_players:
+
+      if aplayer in open_players:
+         aplayer.send(open_players)
+
+         if self == aplayer:
+            for awatcher in self._watchers:
+               if awatcher._can_see_detail:
+                  awatcher.send(open_buf)
+               else:
+                  awatcher.send(wrap_buf)
+         else:
+            self._game.send(aplayer.side)
+      else:
+         self._game.send(aplayer.side, wrap_buf)

test/simple_source/bug31/04_def_annotate.py

@@ -9,7 +9,7 @@ def open(file, mode = "r", buffering = None,
          newline = None, closefd = True) -> "IOBase":
     return text
 
-def foo(x: 'an argument that defaults to 5' = 5):
+def foo1(x: 'an argument that defaults to 5' = 5):
     print(x)
 
 def div(a: dict(type=float, help='the dividend'),

test/simple_source/bug35/04_CALL_FUNCTION_VAR_KW.py

@@ -1,4 +1,5 @@
 # sql/schema.py
+# Note that kwargs comes before "positional" args
 def tometadata(self, metadata, schema, Table, args, name=None):
     table = Table(
         name, metadata, schema=schema,

test/simple_source/bug35/04_importlist.py

@@ -0,0 +1,7 @@
+# Had bug in 3.x in not having semantic importlist rule
+def main(osp, Mfile, mainpyfile, dbg=None):
+    try:
+        from xdis import load_module, PYTHON_VERSION, IS_PYPY
+        return PYTHON_VERSION, IS_PYPY, load_module
+    except:
+        pass

test/simple_source/bug36/10_extended_arg_loop.py

@@ -0,0 +1,49 @@
+# Bug in 3.6 has to do with parsing jumps where
+# the offset is more than 256 bytes so an EXTENDED_ARG
+# instruction is inserted. find_jump_targets() and
+# detect_control_flow need to be able to work in the presence
+# of EXTENDED_ARG.
+
+# This is a problem theoretically in Python before 3.6
+# but since offsets are very large it isn't noticed.
+
+# Code is simplified from trepan2/trepan/cli.py
+import sys
+def main(dbg=None, sys_argv=list(sys.argv)):
+
+    if sys_argv:
+        mainpyfile = None
+    else:
+        mainpyfile = "10"
+        sys.path[0] = "20"
+
+    while True:
+        try:
+            if dbg.program_sys_argv and mainpyfile:
+                normal_termination = dbg.run_script(mainpyfile)
+                if not normal_termination: break
+            else:
+                dbg.core.execution_status = 'No program'
+                dbg.core.processor.process_commands()
+                pass
+
+            dbg.core.execution_status = 'Terminated'
+            dbg.intf[-1].msg("The program finished - quit or restart")
+            dbg.core.processor.process_commands()
+        except IOError:
+            break
+        except RuntimeError:
+            dbg.core.execution_status = 'Restart requested'
+            if dbg.program_sys_argv:
+                sys.argv = list(dbg.program_sys_argv)
+                part1 = ('Restarting %s with arguments:' %
+                         dbg.core.filename(mainpyfile))
+                args  = ' '.join(dbg.program_sys_argv[1:])
+                dbg.intf[-1].msg(args + part1)
+            else: break
+        except SystemExit:
+            break
+        pass
+
+    sys.argv = 5
+    return

test/test_pyenvlib.py

@@ -19,8 +19,6 @@ Step 2: Run the test:
 	  test_pyenvlib --mylib --verify # decompile verify 'mylib'
 """
 
-from __future__ import print_function
-
 from uncompyle6 import main, PYTHON3
 import os, time, shutil, sys
 from fnmatch import fnmatch
@@ -29,7 +27,7 @@ from fnmatch import fnmatch
 
 TEST_VERSIONS=('2.3.7', '2.4.6', '2.5.6', '2.6.9',
                'pypy-2.4.0', 'pypy-2.6.1',
-               'pypy-5.0.1', 'pypy-5.3.1',
+               'pypy-5.0.1', 'pypy-5.3.1', 'pypy3.5-5.7.1-beta',
                '2.7.10', '2.7.11', '2.7.12', '2.7.13',
                '3.0.1', '3.1.5', '3.2.6',
                '3.3.5', '3.3.6',

test/test_pythonlib.py

@@ -27,8 +27,6 @@ Step 2: Run the test:
   test_pythonlib.py --mylib --verify # decompile verify 'mylib'
 """
 
-from __future__ import print_function
-
 import getopt, os, py_compile, sys, shutil, tempfile, time
 
 from uncompyle6 import PYTHON_VERSION
@@ -127,8 +125,10 @@ def do_tests(src_dir, obj_patterns, target_dir, opts):
     if opts['do_compile']:
         compiled_version = opts['compiled_version']
         if compiled_version and PYTHON_VERSION != compiled_version:
-            print("Not compiling: desired Python version is %s but we are running %s" %
-                  (compiled_version, PYTHON_VERSION), file=sys.stderr)
+            sys.stderr.write("Not compiling: "
+                             "desired Python version is %s "
+                             "but we are running %s" %
+                             (compiled_version, PYTHON_VERSION))
         else:
             for root, dirs, basenames in os.walk(src_dir):
                 file_matches(files, root, basenames, PY)
@@ -146,8 +146,8 @@ def do_tests(src_dir, obj_patterns, target_dir, opts):
         file_matches(files, dirname, basenames, obj_patterns)
 
     if not files:
-        print("Didn't come up with any files to test! Try with --compile?",
-              file=sys.stderr)
+        sys.stderr.write("Didn't come up with any files to test! "
+                         "Try with --compile?")
         exit(1)
 
     os.chdir(cwd)
@@ -161,21 +161,21 @@ def do_tests(src_dir, obj_patterns, target_dir, opts):
         except ValueError:
             pass
 
-    print(time.ctime())
-    print('Source directory: ', src_dir)
-    print('Output directory: ', target_dir)
+    print time.ctime()
+    print 'Source directory: ', src_dir
+    print 'Output directory: ', target_dir
     try:
         _, _, failed_files, failed_verify = \
           main(src_dir, target_dir, files, [],
                do_verify=opts['do_verify'])
         if failed_files != 0:
-            exit(2)
+            sys.exit(2)
         elif failed_verify != 0:
-            exit(3)
+            sys.exit(3)
 
     except (KeyboardInterrupt, OSError):
         print()
-        exit(1)
+        sys.exit(1)
     if test_opts['rmtree']:
         parent_dir = os.path.dirname(target_dir)
         print("Everything good, removing %s" % parent_dir)
@@ -236,14 +236,13 @@ if __name__ == '__main__':
         if os.path.isdir(src_dir):
             checked_dirs.append([src_dir, pattern, target_dir])
         else:
-            print("Can't find directory %s. Skipping" % src_dir,
-                  file=sys.stderr)
+            sys.stderr.write("Can't find directory %s. Skipping" % src_dir)
             continue
         last_compile_version = compiled_version
         pass
 
     if not checked_dirs:
-        print("No directories found to check", file=sys.stderr)
+        sys.stderr.write("No directories found to check\n")
         sys.exit(1)
 
     test_opts['compiled_version'] = last_compile_version

uncompyle6/bin/pydisassemble.py

@@ -3,7 +3,6 @@
 #
 # Copyright (c) 2015-2016 by Rocky Bernstein <rb@dustyfeet.com>
 #
-from __future__ import print_function
 import sys, os, getopt
 
 from uncompyle6.disas import disassemble_file
@@ -26,7 +25,7 @@ Options:
   -V | --version     show version and stop
   -h | --help        show this message
 
-""".format(program)
+""" % (program, program)
 
 PATTERNS = ('*.pyc', '*.pyo')
 
@@ -37,15 +36,15 @@ Type -h for for full help.""" % program
     native = True
 
     if len(sys.argv) == 1:
-        print("No file(s) given", file=sys.stderr)
-        print(Usage_short, file=sys.stderr)
+        sys.stderr.write("No file(s) given\n")
+        sys.stderr.write(Usage_short)
         sys.exit(1)
 
     try:
         opts, files = getopt.getopt(sys.argv[1:], 'hVU',
                                     ['help', 'version', 'uncompyle6'])
-    except getopt.GetoptError as e:
-        print('%s: %s' % (os.path.basename(sys.argv[0]), e),  file=sys.stderr)
+    except getopt.GetoptError(e):
+        sys.stderr.write('%s: %s' % (os.path.basename(sys.argv[0]), e))
         sys.exit(-1)
 
     for opt, val in opts:
@@ -59,15 +58,14 @@ Type -h for for full help.""" % program
             native = False
         else:
             print(opt)
-            print(Usage_short, file=sys.stderr)
+            sys.stderr.write(Usage_short)
             sys.exit(1)
 
     for file in files:
         if os.path.exists(files[0]):
             disassemble_file(file, sys.stdout, native)
         else:
-            print("Can't read %s - skipping" % files[0],
-                  file=sys.stderr)
+            sys.stderr.write("Can't read %s - skipping\n" % files[0])
             pass
         pass
     return

uncompyle6/bin/uncompile.py

@@ -1,13 +1,12 @@
 #!/usr/bin/env python
 # Mode: -*- python -*-
 #
-# Copyright (c) 2015-2016 by Rocky Bernstein
+# Copyright (c) 2015-2017 by Rocky Bernstein
 # Copyright (c) 2000-2002 by hartmut Goebel <h.goebel@crazy-compilers.com>
 #
-from __future__ import print_function
 import sys, os, getopt, time
 
-program, ext = os.path.splitext(os.path.basename(__file__))
+program = 'uncompyle6'
 
 __doc__ = """
 Usage:
@@ -65,11 +64,11 @@ def usage():
 
 
 def main_bin():
-    if not (sys.version_info[0:2] in ((2, 6), (2, 7),
-                                      (3, 1), (3, 2), (3, 3),
+    if not (sys.version_info[0:2] in ((2, 4), (2, 5), (2, 6), (2, 7),
+                                      (3, 2), (3, 3),
                                       (3, 4), (3, 5), (3, 6))):
-        print('Error: %s requires Python 2.6-2.7, or 3.1-3.6' % program,
-              file=sys.stderr)
+        sys.stderr.write('Error: %s requires Python 2.4 2.5 2.6, 2.7, '
+                         '3.2, 3.3, 3.4, 3.5, or 3.6' % program)
         sys.exit(-1)
 
     do_verify = recurse_dirs = False
@@ -84,8 +83,8 @@ def main_bin():
         opts, files = getopt.getopt(sys.argv[1:], 'hagtdrVo:c:p:',
                                     'help asm grammar linemaps recurse timestamp tree '
                                     'verify version showgrammar'.split(' '))
-    except getopt.GetoptError as e:
-        print('%s: %s' % (os.path.basename(sys.argv[0]), e),  file=sys.stderr)
+    except getopt.GetoptError(e):
+        sys.stderr.write('%s: %s\n' % (os.path.basename(sys.argv[0]), e))
         sys.exit(-1)
 
     options = {}
@@ -119,7 +118,7 @@ def main_bin():
         elif opt in ('--recurse', '-r'):
             recurse_dirs = True
         else:
-            print(opt, file=sys.stderr)
+            sys.stderr.write(opt)
             usage()
 
     # expand directory if specified
@@ -144,7 +143,7 @@ def main_bin():
         files = [f[sb_len:] for f in files]
 
     if not files:
-        print("No files given", file=sys.stderr)
+        sys.stderr.write("No files given\n")
         usage()
 
     if outfile == '-':
@@ -175,7 +174,7 @@ def main_bin():
         try:
             from Queue import Empty
         except ImportError:
-            from Queue import Empty
+            from queue import Empty
 
         fqueue = Queue(len(files)+numproc)
         for f in files:

uncompyle6/disas.py

@@ -16,8 +16,6 @@ Second, we need structured instruction information for the
 want to run on Python 2.7.
 """
 
-from __future__ import print_function
-
 import sys
 from collections import deque
 
@@ -37,10 +35,9 @@ def disco(version, co, out=None, is_pypy=False):
 
     # store final output stream for case of error
     real_out = out or sys.stdout
-    print('# Python %s' % version, file=real_out)
+    real_out.write('# Python %s\n' % version)
     if co.co_filename:
-        print('# Embedded file name: %s' % co.co_filename,
-              file=real_out)
+        real_out.write('# Embedded file name: %s\n' % co.co_filename)
 
     scanner = get_scanner(version, is_pypy=is_pypy)
 
@@ -52,16 +49,15 @@ def disco_loop(disasm, queue, real_out):
     while len(queue) > 0:
         co = queue.popleft()
         if co.co_name != '<module>':
-            print('\n# %s line %d of %s' %
-                      (co.co_name, co.co_firstlineno, co.co_filename),
-                      file=real_out)
+            real_out.write('\n# %s line %d of %s\n' %
+                      (co.co_name, co.co_firstlineno, co.co_filename))
         tokens, customize = disasm(co)
         for t in tokens:
             if iscode(t.pattr):
                 queue.append(t.pattr)
             elif iscode(t.attr):
                 queue.append(t.attr)
-            print(t, file=real_out)
+            real_out.write(t)
             pass
         pass
 

uncompyle6/linenumbers.py

@@ -10,7 +10,7 @@ def line_number_mapping(pyc_filename, src_filename):
      source_size) = load_module(pyc_filename)
     try:
         code2 = load_file(src_filename)
-    except SyntaxError as e:
+    except SyntaxError, e:
         return str(e)
 
     queue = deque([code1, code2])

uncompyle6/main.py

@@ -1,4 +1,3 @@
-from __future__ import print_function
 import datetime, os, subprocess, sys, tempfile
 
 from uncompyle6 import verify, IS_PYPY
@@ -22,31 +21,36 @@ def decompile(
 
     # store final output stream for case of error
     real_out = out or sys.stdout
-    co_pypy_str = 'PyPy ' if is_pypy else ''
-    run_pypy_str = 'PyPy ' if IS_PYPY else ''
-    print('# uncompyle6 version %s\n'
-          '# %sPython bytecode %s%s\n# Decompiled from: %sPython %s' %
-          (VERSION, co_pypy_str, bytecode_version,
-           " (%d)" % magic_int if magic_int else "",
-           run_pypy_str, '\n# '.join(sys.version.split('\n'))),
-           file=real_out)
-    if co.co_filename:
-        print('# Embedded file name: %s' % co.co_filename,
-              file=real_out)
-    if timestamp:
-        print('# Compiled at: %s' % datetime.datetime.fromtimestamp(timestamp),
-              file=real_out)
-    if source_size:
-        print('# Size of source mod 2**32: %d bytes' % source_size,
-               file=real_out)
+    if is_pypy:
+        co_pypy_str = 'PyPy '
+    else:
+        co_pypy_str = ''
 
-    try:
-        pysource.deparse_code(bytecode_version, co, out, showasm, showast,
-                              showgrammar, code_objects=code_objects,
-                              is_pypy=is_pypy)
-    except pysource.SourceWalkerError as e:
-        # deparsing failed
-        raise pysource.SourceWalkerError(str(e))
+    if IS_PYPY:
+        run_pypy_str = 'PyPy '
+    else:
+        run_pypy_str = ''
+
+    if magic_int:
+        m = str(magic_int)
+    else:
+        m = ""
+    real_out.write('# uncompyle6 version %s\n'
+          '# %sPython bytecode %s%s\n# Decompiled from: %sPython %s\n' %
+          (VERSION, co_pypy_str, bytecode_version,
+           " (%s)" % m, run_pypy_str,
+           '\n# '.join(sys.version.split('\n'))))
+    if co.co_filename:
+        real_out.write('# Embedded file name: %s\n' % co.co_filename)
+    if timestamp:
+        real_out.write('# Compiled at: %s\n' %
+                       datetime.datetime.fromtimestamp(timestamp))
+    if source_size:
+        real_out.write('# Size of source mod 2**32: %d bytes\n' % source_size)
+
+    pysource.deparse_code(bytecode_version, co, out, showasm, showast,
+                          showgrammar, code_objects=code_objects,
+                          is_pypy=is_pypy)
 
 # For compatiblity
 uncompyle = decompile
@@ -128,7 +132,10 @@ def main(in_base, out_base, files, codes, outfile=None,
                 junk, outfile = tempfile.mkstemp(suffix=".py",
                                              prefix=prefix)
                 # Unbuffer output if possible
-                buffering = -1 if sys.stdout.isatty() else 0
+                if sys.stdout.isatty():
+                    buffering = -1
+                else:
+                    buffering = 0
                 sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', buffering)
                 tee = subprocess.Popen(["tee", outfile], stdin=subprocess.PIPE)
                 os.dup2(tee.stdin.fileno(), sys.stdout.fileno())
@@ -145,9 +152,9 @@ def main(in_base, out_base, files, codes, outfile=None,
         try:
             decompile_file(infile, outstream, showasm, showast, showgrammar)
             tot_files += 1
-        except (ValueError, SyntaxError, ParserError, pysource.SourceWalkerError) as e:
+        except (ValueError, SyntaxError, ParserError, pysource.SourceWalkerError):
             sys.stdout.write("\n")
-            sys.stderr.write("\n# file %s\n# %s\n" % (infile, e))
+            sys.stderr.write("# file %s\n" % (infile))
             failed_files += 1
         except KeyboardInterrupt:
             if outfile:
@@ -181,31 +188,35 @@ def main(in_base, out_base, files, codes, outfile=None,
                         msg = verify.compare_code_with_srcfile(infile, current_outfile, weak_verify=weak_verify)
                         if not current_outfile:
                             if not msg:
-                                print('\n# okay decompiling %s' % infile)
+                                print '\n# okay decompiling %s' % infile
                                 okay_files += 1
                             else:
-                                print('\n# %s\n\t%s', infile, msg)
-                    except verify.VerifyCmpError as e:
+                                print '\n# %s\n\t%s', infile, msg
+                    except verify.VerifyCmpError, e:
                         print(e)
                         verify_failed_files += 1
                         os.rename(current_outfile, current_outfile + '_unverified')
                         sys.stderr.write("### Error Verifying %s\n" % filename)
                         sys.stderr.write(str(e) + "\n")
                         if not outfile:
+                            sys.stder.write("### Error Verifiying %s" %
+                                            filename)
+                            sys.stderr.write(e)
                             if raise_on_error:
                                 raise
                             pass
                         pass
                 pass
             elif do_verify:
-                sys.stderr.write("\n### uncompile successful, but no file to compare against\n")
+                sys.stderr.write("\n### uncompile successful, "
+                                 "but no file to compare against")
                 pass
             else:
                 okay_files += 1
                 if not current_outfile:
                     mess = '\n# okay decompiling'
                     # mem_usage = __memUsage()
-                    print(mess, infile)
+                    print mess, infile
         if current_outfile:
             sys.stdout.write("%s\r" %
                              status_msg(do_verify, tot_files, okay_files, failed_files, verify_failed_files))

uncompyle6/parser.py

@@ -3,18 +3,16 @@
 #  Copyright (c) 2000-2002 by hartmut Goebel <h.goebel@crazy-compilers.com>
 #  Copyright (c) 1999 John Aycock
 """
-Common uncompyle parser routines.
+Common uncompyle6 parser routines.
 """
 
-from __future__ import print_function
-
 import sys
 
 from xdis.code import iscode
+from xdis.magics import py_str2float
 from spark_parser import GenericASTBuilder, DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
 from uncompyle6.show import maybe_show_asm
 
-
 class ParserError(Exception):
     def __init__(self, token, offset):
         self.token = token
@@ -30,13 +28,16 @@ class PythonParser(GenericASTBuilder):
 
     def __init__(self, AST, start, debug):
         super(PythonParser, self).__init__(AST, start, debug)
-        self.collect = frozenset(
-            ['stmts', 'except_stmts', '_stmts', 'load_attrs',
-             'exprlist', 'kvlist', 'kwargs', 'come_froms', '_come_from',
-              # Python < 3
-             'print_items',
-             # PyPy:
-             'kvlist_n'])
+        # FIXME: customize per python parser version
+        nt_list = [
+            'stmts', 'except_stmts', '_stmts', 'load_attrs',
+            'exprlist', 'kvlist', 'kwargs', 'come_froms', '_come_from',
+            'importlist',
+            # Python < 3
+            'print_items',
+            # PyPy:
+            'kvlist_n']
+        self.collect = frozenset(nt_list)
 
     def ast_first_offset(self, ast):
         if hasattr(ast, 'offset'):
@@ -44,21 +45,25 @@ class PythonParser(GenericASTBuilder):
         else:
             return self.ast_first_offset(ast[0])
 
-    def add_unique_rule(self, rule, opname, count, customize):
+    def add_unique_rule(self, rule, opname, arg_count, customize):
         """Add rule to grammar, but only if it hasn't been added previously
-           opname and count are used in the customize() semantic the actions
-           to add the semantic action rule. Often, count is not used.
+           opname and stack_count are used in the customize() semantic
+           the actions to add the semantic action rule. Stack_count is
+           used in custom opcodes like MAKE_FUNCTION to indicate how
+           many arguments it has. Often it is not used.
         """
         if rule not in self.new_rules:
             # print("XXX ", rule) # debug
             self.new_rules.add(rule)
             self.addRule(rule, nop_func)
-            customize[opname] = count
+            customize[opname] = arg_count
             pass
         return
 
     def add_unique_rules(self, rules, customize):
-        """Add rules (a list of string) to grammar
+        """Add rules (a list of string) to grammar. Note that
+        the rules must not be those that set arg_count in the
+        custom dictionary.
         """
         for rule in rules:
             if len(rule) == 0:
@@ -68,7 +73,9 @@ class PythonParser(GenericASTBuilder):
         return
 
     def add_unique_doc_rules(self, rules_str, customize):
-        """Add rules (a docstring-like list of rules) to grammar
+        """Add rules (a docstring-like list of rules) to grammar.
+        Note that the rules must not be those that set arg_count in the
+        custom dictionary.
         """
         rules = [r.strip() for r in rules_str.split("\n")]
         self.add_unique_rules(rules, customize)
@@ -85,14 +92,17 @@ class PythonParser(GenericASTBuilder):
         for i in dir(self):
             setattr(self, i, None)
 
-    def debug_reduce(self, rule, tokens, parent, i):
+    def debug_reduce(self, rule, tokens, parent, last_token_pos):
         """Customized format and print for our kind of tokens
         which gets called in debugging grammar reduce rules
         """
         def fix(c):
             s = str(c)
-            i = s.find('_')
-            return s if i == -1 else s[:i]
+            last_token_pos = s.find('_')
+            if last_token_pos == -1:
+                return s
+            else:
+                return s[:last_token_pos]
 
         prefix = ''
         if parent and tokens:
@@ -104,31 +114,38 @@ class PythonParser(GenericASTBuilder):
             if hasattr(p_token, 'offset'):
                 prefix += "%3s" % fix(p_token.offset)
                 if len(rule[1]) > 1:
-                    prefix += '-%-3s ' % fix(tokens[i-1].offset)
+                    prefix += '-%-3s ' % fix(tokens[last_token_pos-1].offset)
                 else:
                     prefix += '     '
         else:
             prefix = '               '
 
-        print("%s%s ::= %s" % (prefix, rule[0], ' '.join(rule[1])))
+        print("%s%s ::= %s (%d)" % (prefix, rule[0], ' '.join(rule[1]), last_token_pos))
 
     def error(self, instructions, index):
         # Find the last line boundary
+        start, finish = -1, -1
         for start in range(index, -1, -1):
             if instructions[start].linestart:  break
             pass
         for finish in range(index+1, len(instructions)):
             if instructions[finish].linestart:  break
             pass
-        err_token = instructions[index]
-        print("Instruction context:")
-        for i in range(start, finish):
-            indent = '   ' if i != index else '-> '
-            print("%s%s" % (indent, instructions[i]))
-        raise ParserError(err_token, err_token.offset)
+        if start > 0:
+            err_token = instructions[index]
+            print("Instruction context:")
+            for i in range(start, finish):
+                if i != index:
+                    indent = '   '
+                else:
+                    indent = '-> '
+                print "%s%s" % (indent, instructions[i])
+            raise ParserError(err_token, err_token.offset)
+        else:
+            raise ParserError(None, -1)
 
     def typestring(self, token):
-        return token.type
+        return token.kind
 
     def nonterminal(self, nt, args):
         if nt in self.collect and len(args) > 1:
@@ -250,8 +267,15 @@ class PythonParser(GenericASTBuilder):
 
         stmt ::= return_stmt
         return_stmt ::= ret_expr RETURN_VALUE
+        return_stmt_lambda ::= ret_expr RETURN_VALUE_LAMBDA
+
+        # return_stmts are a sequence of statements that ends in a RETURN statement.
+        # In later Python versions with jump optimization, this can cause JUMPs
+        # that would normally appear to be omitted.
+
         return_stmts ::= return_stmt
         return_stmts ::= _stmts return_stmt
+
         """
         pass
 
@@ -385,15 +409,15 @@ class PythonParser(GenericASTBuilder):
         stmt ::= importstar
         stmt ::= importmultiple
 
-        importlist2 ::= importlist2 import_as
-        importlist2 ::= import_as
-        import_as ::= IMPORT_NAME designator
-        import_as ::= IMPORT_NAME load_attrs designator
-        import_as ::= IMPORT_FROM designator
+        importlist ::= importlist import_as
+        importlist ::= import_as
+        import_as  ::= IMPORT_NAME designator
+        import_as  ::= IMPORT_NAME load_attrs designator
+        import_as  ::= IMPORT_FROM designator
 
         importstmt ::= LOAD_CONST LOAD_CONST import_as
         importstar ::= LOAD_CONST LOAD_CONST IMPORT_NAME IMPORT_STAR
-        importfrom ::= LOAD_CONST LOAD_CONST IMPORT_NAME importlist2 POP_TOP
+        importfrom ::= LOAD_CONST LOAD_CONST IMPORT_NAME importlist POP_TOP
         importmultiple ::= LOAD_CONST LOAD_CONST import_as imports_cont
 
         imports_cont ::= imports_cont import_cont
@@ -465,27 +489,24 @@ class PythonParser(GenericASTBuilder):
         expr ::= buildslice3
         expr ::= yield
 
-        # Possibly Python < 2.3
-        # expr ::= SET_LINENO
-
         binary_expr ::= expr expr binary_op
-        binary_op ::= BINARY_ADD
-        binary_op ::= BINARY_MULTIPLY
-        binary_op ::= BINARY_AND
-        binary_op ::= BINARY_OR
-        binary_op ::= BINARY_XOR
-        binary_op ::= BINARY_SUBTRACT
-        binary_op ::= BINARY_TRUE_DIVIDE
-        binary_op ::= BINARY_FLOOR_DIVIDE
-        binary_op ::= BINARY_MODULO
-        binary_op ::= BINARY_LSHIFT
-        binary_op ::= BINARY_RSHIFT
-        binary_op ::= BINARY_POWER
+        binary_op   ::= BINARY_ADD
+        binary_op   ::= BINARY_MULTIPLY
+        binary_op   ::= BINARY_AND
+        binary_op   ::= BINARY_OR
+        binary_op   ::= BINARY_XOR
+        binary_op   ::= BINARY_SUBTRACT
+        binary_op   ::= BINARY_TRUE_DIVIDE
+        binary_op   ::= BINARY_FLOOR_DIVIDE
+        binary_op   ::= BINARY_MODULO
+        binary_op   ::= BINARY_LSHIFT
+        binary_op   ::= BINARY_RSHIFT
+        binary_op   ::= BINARY_POWER
 
-        unary_expr ::= expr unary_op
-        unary_op ::= UNARY_POSITIVE
-        unary_op ::= UNARY_NEGATIVE
-        unary_op ::= UNARY_INVERT
+        unary_expr  ::= expr unary_op
+        unary_op    ::= UNARY_POSITIVE
+        unary_op    ::= UNARY_NEGATIVE
+        unary_op    ::= UNARY_INVERT
 
         unary_not ::= expr UNARY_NOT
 
@@ -526,7 +547,9 @@ class PythonParser(GenericASTBuilder):
         stmt ::= return_lambda
         stmt ::= conditional_lambda
 
-        return_lambda ::= ret_expr RETURN_VALUE LAMBDA_MARKER
+        return_lambda ::= ret_expr RETURN_VALUE_LAMBDA LAMBDA_MARKER
+        return_lambda ::= ret_expr RETURN_VALUE_LAMBDA
+
         conditional_lambda ::= expr jmp_false return_if_stmt return_stmt LAMBDA_MARKER
 
         cmp ::= cmp_list
@@ -605,7 +628,15 @@ def get_python_parser(
     explanation of the different modes.
     """
 
+    # If version is a string, turn that into the corresponding float.
+    if isinstance(version, str):
+        version = py_str2float(version)
+
     # FIXME: there has to be a better way...
+    # We could do this as a table lookup, but that would force us
+    # in import all of the parsers all of the time. Perhaps there is
+    # a lazy way of doing the import?
+
     if version < 3.0:
         if version == 1.5:
             import uncompyle6.parsers.parse15 as parse15
@@ -714,7 +745,7 @@ def get_python_parser(
             else:
                 p = parse3.Python3ParserSingle(debug_parser)
     p.version = version
-    # p.dumpGrammar() # debug
+    # p.dump_grammar() # debug
     return p
 
 class PythonParserSingle(PythonParser):
@@ -758,7 +789,8 @@ def python_parser(version, co, out=sys.stdout, showasm=False,
 if __name__ == '__main__':
     def parse_test(co):
         from uncompyle6 import PYTHON_VERSION, IS_PYPY
+        ast = python_parser('2.7.13', co, showasm=True, is_pypy=True)
         ast = python_parser(PYTHON_VERSION, co, showasm=True, is_pypy=IS_PYPY)
         print(ast)
         return
-    parse_test(parse_test.__code__)
+    # parse_test(parse_test.__code__)

uncompyle6/parsers/astnode.py

@@ -16,7 +16,7 @@ class AST(spark_AST):
         return self.__repr1__('', None)
 
     def __repr1__(self, indent, sibNum=None):
-        rv = str(self.type)
+        rv = str(self.kind)
         if sibNum is not None:
             rv = "%2d. %s" % (sibNum, rv)
         enumerate_children = False

uncompyle6/parsers/parse15.py

@@ -29,8 +29,8 @@ class Python15ParserSingle(Python21Parser, PythonParserSingle):
 if __name__ == '__main__':
     # Check grammar
     p = Python15Parser()
-    p.checkGrammar()
-    p.dumpGrammar()
+    p.check_grammar()
+    p.dump_grammar()
 
 # local variables:
 # tab-width: 4

uncompyle6/parsers/parse2.py

@@ -12,8 +12,6 @@ If we succeed in creating a parse tree, then we have a Python program
 that a later phase can turn into a sequence of ASCII text.
 """
 
-from __future__ import print_function
-
 from uncompyle6.parser import PythonParser, PythonParserSingle, nop_func
 from uncompyle6.parsers.astnode import AST
 from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
@@ -397,6 +395,8 @@ class Python2Parser(PythonParser):
         return
 
     def reduce_is_invalid(self, rule, ast, tokens, first, last):
+        if tokens is None:
+            return False
         lhs = rule[0]
         if lhs in ('augassign1', 'augassign2') and ast[0][0] == 'and':
             return True
@@ -417,4 +417,4 @@ class Python2ParserSingle(Python2Parser, PythonParserSingle):
 if __name__ == '__main__':
     # Check grammar
     p = Python2Parser()
-    p.checkGrammar()
+    p.check_grammar()

uncompyle6/parsers/parse21.py

@@ -33,8 +33,8 @@ class Python21ParserSingle(Python22Parser, PythonParserSingle):
 if __name__ == '__main__':
     # Check grammar
     p = Python21Parser()
-    p.checkGrammar()
-    p.dumpGrammar()
+    p.check_grammar()
+    p.dump_grammar()
 
 # local variables:
 # tab-width: 4

uncompyle6/parsers/parse22.py

@@ -26,8 +26,8 @@ class Python22ParserSingle(Python23Parser, PythonParserSingle):
 if __name__ == '__main__':
     # Check grammar
     p = Python22Parser()
-    p.checkGrammar()
-    p.dumpGrammar()
+    p.check_grammar()
+    p.dump_grammar()
 
 # local variables:
 # tab-width: 4

uncompyle6/parsers/parse23.py

@@ -67,8 +67,8 @@ class Python23ParserSingle(Python23Parser, PythonParserSingle):
 if __name__ == '__main__':
     # Check grammar
     p = Python23Parser()
-    p.checkGrammar()
-    p.dumpGrammar()
+    p.check_grammar()
+    p.dump_grammar()
 
 # local variables:
 # tab-width: 4

uncompyle6/parsers/parse24.py

@@ -27,7 +27,7 @@ class Python24Parser(Python25Parser):
         # keep positions similar to simplify semantic actions
 
         importstmt ::= filler LOAD_CONST import_as
-        importfrom ::= filler LOAD_CONST IMPORT_NAME importlist2 POP_TOP
+        importfrom ::= filler LOAD_CONST IMPORT_NAME importlist POP_TOP
         importstar ::= filler LOAD_CONST IMPORT_NAME IMPORT_STAR
 
         importmultiple ::= filler LOAD_CONST import_as imports_cont
@@ -55,13 +55,14 @@ class Python24Parser(Python25Parser):
         invalid = super(Python24Parser,
                         self).reduce_is_invalid(rule, ast,
                                                 tokens, first, last)
-        if invalid:
+        if invalid or tokens is None:
             return invalid
 
-        # FiXME: this code never gets called...
         lhs = rule[0]
         if lhs == 'nop_stmt':
-            return not int(tokens[first].pattr) == tokens[last].offset
+            l = len(tokens)
+            if 0 <= l < len(tokens):
+                return not int(tokens[first].pattr) == tokens[last].offset
 
         return False
 
@@ -71,4 +72,4 @@ class Python24ParserSingle(Python24Parser, PythonParserSingle):
 if __name__ == '__main__':
     # Check grammar
     p = Python24Parser()
-    p.checkGrammar()
+    p.check_grammar()

uncompyle6/parsers/parse25.py

@@ -60,4 +60,4 @@ class Python25ParserSingle(Python26Parser, PythonParserSingle):
 if __name__ == '__main__':
     # Check grammar
     p = Python25Parser()
-    p.checkGrammar()
+    p.check_grammar()

uncompyle6/parsers/parse26.py

@@ -84,6 +84,12 @@ class Python26Parser(Python2Parser):
         ja_cf_pop ::= JUMP_ABSOLUTE come_froms POP_TOP
         jf_cf_pop ::= JUMP_FORWARD come_froms POP_TOP
 
+        # The first optional COME_FROM when it appears is really
+        # COME_FROM_LOOP, but in <= 2.6 we don't distinguish
+        # this
+
+        cf_jb_cf_pop ::= _come_from JUMP_BACK come_froms POP_TOP
+
         bp_come_from    ::= POP_BLOCK COME_FROM
         jb_bp_come_from ::= JUMP_BACK bp_come_from
 
@@ -111,7 +117,8 @@ class Python26Parser(Python2Parser):
         break_stmt ::= BREAK_LOOP JUMP_BACK
 
         # Semantic actions want else_suitel to be at index 3
-        ifelsestmtl ::= testexpr c_stmts_opt jb_cf_pop else_suitel
+        ifelsestmtl ::= testexpr c_stmts_opt cf_jb_cf_pop else_suitel
+
         ifelsestmtc ::= testexpr c_stmts_opt ja_cf_pop else_suitec
 
         # Semantic actions want suite_stmts_opt to be at index 3
@@ -240,7 +247,9 @@ class Python26Parser(Python2Parser):
         and  ::= expr JUMP_IF_FALSE POP_TOP expr JUMP_IF_FALSE POP_TOP
         cmp_list ::= expr cmp_list1 ROT_TWO COME_FROM POP_TOP _come_from
 
-        conditional_lambda ::= expr jmp_false_then return_if_stmt return_stmt LAMBDA_MARKER
+        return_if_lambda   ::= RETURN_END_IF_LAMBDA POP_TOP
+        conditional_lambda ::= expr jmp_false_then expr return_if_lambda
+                               return_stmt_lambda LAMBDA_MARKER
         """
 
     def add_custom_rules(self, tokens, customize):
@@ -251,7 +260,7 @@ class Python26Parser(Python2Parser):
         invalid = super(Python26Parser,
                         self).reduce_is_invalid(rule, ast,
                                                 tokens, first, last)
-        if invalid:
+        if invalid or tokens is None:
             return invalid
         if rule == ('and', ('expr', 'jmp_false', 'expr', '\\e_come_from_opt')):
             # Test that jmp_false jumps to the end of "and"
@@ -267,10 +276,10 @@ class Python26ParserSingle(Python2Parser, PythonParserSingle):
 if __name__ == '__main__':
     # Check grammar
     p = Python26Parser()
-    p.checkGrammar()
+    p.check_grammar()
     from uncompyle6 import PYTHON_VERSION, IS_PYPY
     if PYTHON_VERSION == 2.6:
-        lhs, rhs, tokens, right_recursive = p.checkSets()
+        lhs, rhs, tokens, right_recursive = p.check_sets()
         from uncompyle6.scanner import get_scanner
         s = get_scanner(PYTHON_VERSION, IS_PYPY)
         opcode_set = set(s.opc.opname).union(set(

uncompyle6/parsers/parse27.py

@@ -94,6 +94,10 @@ class Python27Parser(Python2Parser):
                 WITH_CLEANUP END_FINALLY
 
         # Common with 2.6
+        return_if_lambda   ::= RETURN_END_IF_LAMBDA COME_FROM
+        conditional_lambda ::= expr jmp_false expr return_if_lambda
+                               return_stmt_lambda LAMBDA_MARKER
+
         while1stmt ::= SETUP_LOOP return_stmts bp_come_from
         while1stmt ::= SETUP_LOOP return_stmts COME_FROM
         """
@@ -125,10 +129,10 @@ class Python27ParserSingle(Python27Parser, PythonParserSingle):
 if __name__ == '__main__':
     # Check grammar
     p = Python27Parser()
-    p.checkGrammar()
+    p.check_grammar()
     from uncompyle6 import PYTHON_VERSION, IS_PYPY
     if PYTHON_VERSION == 2.7:
-        lhs, rhs, tokens, right_recursive = p.checkSets()
+        lhs, rhs, tokens, right_recursive = p.check_sets()
         from uncompyle6.scanner import get_scanner
         s = get_scanner(PYTHON_VERSION, IS_PYPY)
         opcode_set = set(s.opc.opname).union(set(
@@ -144,4 +148,5 @@ if __name__ == '__main__':
                              for t in remain_tokens])
         remain_tokens = set(remain_tokens) - opcode_set
         print(remain_tokens)
-        # p.dumpGrammar()
+        p.check_grammar()
+        p.dump_grammar()

uncompyle6/parsers/parse3.py

@@ -15,11 +15,10 @@ If we succeed in creating a parse tree, then we have a Python program
 that a later phase can turn into a sequence of ASCII text.
 """
 
-from __future__ import print_function
-
 from uncompyle6.parser import PythonParser, PythonParserSingle, nop_func
 from uncompyle6.parsers.astnode import AST
 from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
+from xdis import PYTHON3
 
 class Python3Parser(PythonParser):
 
@@ -156,8 +155,13 @@ class Python3Parser(PythonParser):
         # of missing "else" clauses. Therefore we include grammar
         # rules with and without ELSE.
 
-        ifelsestmt ::= testexpr c_stmts_opt JUMP_FORWARD else_suite opt_come_from_except
-        ifelsestmt ::= testexpr c_stmts_opt jump_forward_else else_suite _come_from
+        ifelsestmt ::= testexpr c_stmts_opt JUMP_FORWARD
+                       else_suite opt_come_from_except
+        ifelsestmt ::= testexpr c_stmts_opt jump_forward_else
+                       else_suite _come_from
+
+        # ifelsestmt ::= testexpr c_stmts_opt jump_forward_else
+        #                passstmt  _come_from
 
         ifelsestmtc ::= testexpr c_stmts_opt JUMP_ABSOLUTE else_suitec
         ifelsestmtc ::= testexpr c_stmts_opt jump_absolute_else else_suitec
@@ -253,8 +257,14 @@ class Python3Parser(PythonParser):
                 POP_BLOCK LOAD_CONST COME_FROM_WITH
                 WITH_CLEANUP END_FINALLY
 
+        ## FIXME: Right now we have erroneous jump targets
+        ## This below is probably not correct when the COME_FROM is put in the right place
         and ::= expr jmp_false expr COME_FROM
         or  ::= expr jmp_true  expr COME_FROM
+
+        # # something like the below is needed when the jump targets are fixed
+        ## or  ::= expr JUMP_IF_TRUE_OR_POP COME_FROM expr
+        ## and ::= expr JUMP_IF_FALSE_OR_POP COME_FROM expr
         '''
 
     def p_misc3(self, args):
@@ -417,6 +427,13 @@ class Python3Parser(PythonParser):
         # a JUMP_ABSOLUTE with no COME_FROM
         conditional    ::= expr jmp_false expr jump_absolute_else expr
 
+        return_if_lambda   ::= RETURN_END_IF_LAMBDA
+        conditional_lambda ::= expr jmp_false return_stmt_lambda
+                               return_stmt_lambda LAMBDA_MARKER
+        conditional_lambda ::= expr jmp_false expr return_if_lambda
+                               return_stmt_lambda LAMBDA_MARKER
+
+
         expr ::= LOAD_CLASSNAME
 
         # Python 3.4+
@@ -427,7 +444,7 @@ class Python3Parser(PythonParser):
     @staticmethod
     def call_fn_name(token):
         """Customize CALL_FUNCTION to add the number of positional arguments"""
-        return '%s_%i' % (token.type, token.attr)
+        return '%s_%i' % (token.kind, token.attr)
 
     def custom_build_class_rule(self, opname, i, token, tokens, customize):
         '''
@@ -443,16 +460,16 @@ class Python3Parser(PythonParser):
         # FIXME: I bet this can be simplified
         # look for next MAKE_FUNCTION
         for i in range(i+1, len(tokens)):
-            if tokens[i].type.startswith('MAKE_FUNCTION'):
+            if tokens[i].kind.startswith('MAKE_FUNCTION'):
                 break
-            elif tokens[i].type.startswith('MAKE_CLOSURE'):
+            elif tokens[i].kind.startswith('MAKE_CLOSURE'):
                 break
             pass
         assert i < len(tokens), "build_class needs to find MAKE_FUNCTION or MAKE_CLOSURE"
-        assert tokens[i+1].type == 'LOAD_CONST', \
+        assert tokens[i+1].kind == 'LOAD_CONST', \
           "build_class expecting CONST after MAKE_FUNCTION/MAKE_CLOSURE"
         for i in range(i, len(tokens)):
-            if tokens[i].type == 'CALL_FUNCTION':
+            if tokens[i].kind == 'CALL_FUNCTION':
                 call_fn_tok = tokens[i]
                 break
         assert call_fn_tok, "build_class custom rule needs to find CALL_FUNCTION"
@@ -493,11 +510,11 @@ class Python3Parser(PythonParser):
         # Yes, this computation based on instruction name is a little bit hoaky.
         nak = ( len(opname)-len('CALL_FUNCTION') ) // 3
 
-        token.type = self.call_fn_name(token)
+        token.kind = self.call_fn_name(token)
         uniq_param = args_kw + args_pos
         if self.version == 3.5 and opname.startswith('CALL_FUNCTION_VAR'):
-            # Python 3.5 changes the stack position of where * args, the
-            # first LOAD_FAST, below are located.
+            # Python 3.5 changes the stack position of *args. KW args come
+            # after *args.
             # Python 3.6+ replaces CALL_FUNCTION_VAR_KW with CALL_FUNCTION_EX
             if opname.endswith('KW'):
                 kw = 'expr '
@@ -505,33 +522,42 @@ class Python3Parser(PythonParser):
                 kw = ''
             rule = ('call_function ::= expr expr ' +
                     ('pos_arg ' * args_pos) +
-                    ('kwarg ' * args_kw) + kw + token.type)
-            self.add_unique_rule(rule, token.type, uniq_param, customize)
+                    ('kwarg ' * args_kw) + kw + token.kind)
+            self.add_unique_rule(rule, token.kind, uniq_param, customize)
+        if self.version >= 3.6 and opname == 'CALL_FUNCTION_EX_KW':
+            rule = ('call_function36 ::= '
+                    'expr build_tuple_unpack_with_call build_map_unpack_with_call '
+                    'CALL_FUNCTION_EX_KW_1')
+            self.add_unique_rule(rule, token.kind, uniq_param, customize)
+            rule = 'call_function ::= call_function36'
+        else:
+            rule = ('call_function ::= expr ' +
+                    ('pos_arg ' * args_pos) +
+                    ('kwarg ' * args_kw) +
+                    'expr ' * nak + token.kind)
 
-        rule = ('call_function ::= expr ' +
-                ('pos_arg ' * args_pos) +
-                ('kwarg ' * args_kw) +
-                'expr ' * nak + token.type)
-
-        self.add_unique_rule(rule, token.type, uniq_param, customize)
+        self.add_unique_rule(rule, token.kind, uniq_param, customize)
         if self.version >= 3.5:
             rule = ('async_call_function ::= expr ' +
                     ('pos_arg ' * args_pos) +
                     ('kwarg ' * args_kw) +
-                    'expr ' * nak + token.type +
+                    'expr ' * nak + token.kind +
                     ' GET_AWAITABLE LOAD_CONST YIELD_FROM')
-            self.add_unique_rule(rule, token.type, uniq_param, customize)
-            self.add_unique_rule('expr ::= async_call_function', token.type, uniq_param, customize)
+            self.add_unique_rule(rule, token.kind, uniq_param, customize)
+            self.add_unique_rule('expr ::= async_call_function', token.kind, uniq_param, customize)
 
         rule = ('classdefdeco2 ::= LOAD_BUILD_CLASS mkfunc %s%s_%d'
                 %  (('expr ' * (args_pos-1)), opname, args_pos))
-        self.add_unique_rule(rule, token.type, uniq_param, customize)
+        self.add_unique_rule(rule, token.kind, uniq_param, customize)
 
     def add_make_function_rule(self, rule, opname, attr, customize):
         """Python 3.3 added a an addtional LOAD_CONST before MAKE_FUNCTION and
         this has an effect on many rules.
         """
-        new_rule = rule % (('LOAD_CONST ') * (1 if  self.version >= 3.3 else 0))
+        if self.version >= 3.3:
+            new_rule = rule % (('LOAD_CONST ') * 1)
+        else:
+            new_rule = rule % (('LOAD_CONST ') * 0)
         self.add_unique_rule(new_rule, opname, attr, customize)
 
     def add_custom_rules(self, tokens, customize):
@@ -599,7 +625,7 @@ class Python3Parser(PythonParser):
             call_function ::= expr CALL_METHOD
         """
         for i, token in enumerate(tokens):
-            opname = token.type
+            opname = token.kind
             opname_base = opname[:opname.rfind('_')]
 
             if opname == 'PyPy':
@@ -610,9 +636,9 @@ class Python3Parser(PythonParser):
                     assign2_pypy ::= expr expr designator designator
                 """, nop_func)
                 continue
-            elif opname in ('CALL_FUNCTION', 'CALL_FUNCTION_VAR',
-                            'CALL_FUNCTION_VAR_KW') \
-                 or opname.startswith('CALL_FUNCTION_KW'):
+            elif (opname in ('CALL_FUNCTION', 'CALL_FUNCTION_VAR',
+                             'CALL_FUNCTION_VAR_KW', 'CALL_FUNCTION_EX_KW')
+                  or opname.startswith('CALL_FUNCTION_KW')):
                 self.custom_classfunc_rule(opname, token, customize)
             elif opname == 'LOAD_DICTCOMP':
                 rule_pat = ("dictcomp ::= LOAD_DICTCOMP %sMAKE_FUNCTION_0 expr "
@@ -633,6 +659,18 @@ class Python3Parser(PythonParser):
                 self.add_unique_rule(rule, opname, token.attr, customize)
                 rule = 'expr ::= build_list_unpack'
                 self.add_unique_rule(rule, opname, token.attr, customize)
+            elif opname.startswith('BUILD_TUPLE_UNPACK_WITH_CALL'):
+                v = token.attr
+                rule = ('build_tuple_unpack_with_call ::= ' + 'expr1024 ' * int(v//1024) +
+                        'expr32 ' * int((v//32) % 32) +
+                        'expr ' * (v % 32) + opname)
+                self.add_unique_rule(rule, opname, token.attr, customize)
+            elif opname.startswith('BUILD_MAP_UNPACK_WITH_CALL'):
+                v = token.attr
+                rule = ('build_map_unpack_with_call ::= ' + 'expr1024 ' * int(v//1024) +
+                        'expr32 ' * int((v//32) % 32) +
+                        'expr ' * (v % 32) + opname)
+                self.add_unique_rule(rule, opname, token.attr, customize)
             elif opname_base in ('BUILD_LIST', 'BUILD_TUPLE', 'BUILD_SET'):
                 v = token.attr
                 rule = ('build_list ::= ' + 'expr1024 ' * int(v//1024) +
@@ -642,7 +680,10 @@ class Python3Parser(PythonParser):
                 if opname_base == 'BUILD_TUPLE':
                     rule = ('load_closure ::= %s%s' % (('LOAD_CLOSURE ' * v), opname))
                     self.add_unique_rule(rule, opname, token.attr, customize)
-
+                    rule = ('build_tuple ::= ' + 'expr1024 ' * int(v//1024) +
+                        'expr32 ' * int((v//32) % 32) +
+                        'expr ' * (v % 32) + opname)
+                    self.add_unique_rule(rule, opname, token.attr, customize)
             elif opname == 'LOOKUP_METHOD':
                 # A PyPy speciality - DRY with parse2
                 self.add_unique_rule("load_attr ::= expr LOOKUP_METHOD",
@@ -868,7 +909,11 @@ class Python3Parser(PythonParser):
         elif lhs == 'annotate_tuple':
             return not isinstance(tokens[first].attr, tuple)
         elif lhs == 'kwarg':
-            return not isinstance(tokens[first].attr, str)
+            arg = tokens[first].attr
+            if PYTHON3:
+                return not isinstance(arg, str)
+            else:
+                return not (isinstance(arg, str) or isinstance(arg, unicode))
         elif lhs == 'while1elsestmt':
             # if SETUP_LOOP target spans the else part, then this is
             # not while1else. Also do for whileTrue?
@@ -877,7 +922,8 @@ class Python3Parser(PythonParser):
                 last += 1
             return tokens[first].attr == tokens[last].offset
         elif lhs == 'while1stmt':
-            if tokens[last] in ('COME_FROM_LOOP', 'JUMP_BACK'):
+            if (0 <= last < len(tokens)
+                and tokens[last] in ('COME_FROM_LOOP', 'JUMP_BACK')):
                 # jump_back should be right afer SETUP_LOOP. Test?
                 last += 1
             while last < len(tokens) and isinstance(tokens[last].offset, str):
@@ -921,10 +967,10 @@ def info(args):
             p = Python32Parser()
         elif arg == '3.0':
             p = Python30Parser()
-    p.checkGrammar()
+    p.check_grammar()
     if len(sys.argv) > 1 and sys.argv[1] == 'dump':
         print('-' * 50)
-        p.dumpGrammar()
+        p.dump_grammar()
 
 if __name__ == '__main__':
     import sys

uncompyle6/parsers/parse30.py

@@ -2,7 +2,6 @@
 """
 spark grammar differences over Python 3.1 for Python 3.0.
 """
-from __future__ import print_function
 
 from uncompyle6.parser import PythonParserSingle
 from uncompyle6.parsers.parse31 import Python31Parser

uncompyle6/parsers/parse31.py

@@ -2,7 +2,6 @@
 """
 spark grammar differences over Python 3.2 for Python 3.1.
 """
-from __future__ import print_function
 
 from uncompyle6.parser import PythonParserSingle
 from uncompyle6.parsers.parse32 import Python32Parser

uncompyle6/parsers/parse32.py

@@ -2,8 +2,6 @@
 """
 spark grammar differences over Python 3 for Python 3.2.
 """
-from __future__ import print_function
-
 from uncompyle6.parser import PythonParserSingle
 from uncompyle6.parsers.parse3 import Python3Parser
 
@@ -44,7 +42,7 @@ class Python32Parser(Python3Parser):
     def add_custom_rules(self, tokens, customize):
         super(Python32Parser, self).add_custom_rules(tokens, customize)
         for i, token in enumerate(tokens):
-            opname = token.type
+            opname = token.kind
             if opname.startswith('MAKE_FUNCTION_A'):
                 args_pos, args_kw, annotate_args  = token.attr
                 # Check that there are 2 annotated params?

uncompyle6/parsers/parse33.py

@@ -2,7 +2,6 @@
 """
 spark grammar differences over Python 3.2 for Python 3.3.
 """
-from __future__ import print_function
 
 from uncompyle6.parser import PythonParserSingle
 from uncompyle6.parsers.parse32 import Python32Parser

uncompyle6/parsers/parse34.py

@@ -29,10 +29,10 @@ class Python34ParserSingle(Python34Parser, PythonParserSingle):
 if __name__ == '__main__':
     # Check grammar
     p = Python34Parser()
-    p.checkGrammar()
+    p.check_grammar()
     from uncompyle6 import PYTHON_VERSION, IS_PYPY
     if PYTHON_VERSION == 3.4:
-        lhs, rhs, tokens, right_recursive = p.checkSets()
+        lhs, rhs, tokens, right_recursive = p.check_sets()
         from uncompyle6.scanner import get_scanner
         s = get_scanner(PYTHON_VERSION, IS_PYPY)
         opcode_set = set(s.opc.opname).union(set(

uncompyle6/parsers/parse35.py

@@ -2,7 +2,6 @@
 """
 spark grammar differences over Python 3.4 for Python 3.5.
 """
-from __future__ import print_function
 
 from uncompyle6.parser import PythonParserSingle
 from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
@@ -143,7 +142,7 @@ class Python35Parser(Python34Parser):
     def add_custom_rules(self, tokens, customize):
         super(Python35Parser, self).add_custom_rules(tokens, customize)
         for i, token in enumerate(tokens):
-            opname = token.type
+            opname = token.kind
             if opname == 'BUILD_MAP_UNPACK_WITH_CALL':
                 nargs = token.attr % 256
                 map_unpack_n = "map_unpack_%s" % nargs
@@ -153,7 +152,7 @@ class Python35Parser(Python34Parser):
                 self.add_unique_rule(rule, opname, token.attr, customize)
                 call_token = tokens[i+1]
                 if self.version == 3.5:
-                    rule = 'call_function ::= expr unmapexpr ' + call_token.type
+                    rule = 'call_function ::= expr unmapexpr ' + call_token.kind
                     self.add_unique_rule(rule, opname, token.attr, customize)
                 pass
             pass
@@ -165,10 +164,10 @@ class Python35ParserSingle(Python35Parser, PythonParserSingle):
 if __name__ == '__main__':
     # Check grammar
     p = Python35Parser()
-    p.checkGrammar()
+    p.check_grammar()
     from uncompyle6 import PYTHON_VERSION, IS_PYPY
     if PYTHON_VERSION == 3.5:
-        lhs, rhs, tokens, right_recursive = p.checkSets()
+        lhs, rhs, tokens, right_recursive = p.check_sets()
         from uncompyle6.scanner import get_scanner
         s = get_scanner(PYTHON_VERSION, IS_PYPY)
         opcode_set = set(s.opc.opname).union(set(

uncompyle6/parsers/parse36.py

@@ -2,7 +2,6 @@
 """
 spark grammar differences over Python 3.5 for Python 3.6.
 """
-from __future__ import print_function
 
 from uncompyle6.parser import PythonParserSingle
 from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
@@ -25,18 +24,56 @@ class Python36Parser(Python35Parser):
 
         func_args36   ::= expr BUILD_TUPLE_0
         call_function ::= func_args36 unmapexpr CALL_FUNCTION_EX
+        call_function ::= func_args36 build_map_unpack_with_call CALL_FUNCTION_EX_KW_1
 
         withstmt ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK LOAD_CONST
                      WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
 
         call_function ::= expr expr CALL_FUNCTION_EX
-        call_function ::= expr expr expr CALL_FUNCTION_EX_KW
+        call_function ::= expr expr expr CALL_FUNCTION_EX_KW_1
+
+        # This might be valid in < 3.6
+        and  ::= expr jmp_false expr
+
+        # Adds a COME_FROM_ASYNC_WITH over 3.5
+        # FIXME: remove corresponding rule for 3.5?
+        async_with_as_stmt ::= expr
+                               BEFORE_ASYNC_WITH GET_AWAITABLE LOAD_CONST YIELD_FROM
+                               SETUP_ASYNC_WITH designator
+                               suite_stmts_opt
+                               POP_BLOCK LOAD_CONST
+                               COME_FROM_ASYNC_WITH
+                               WITH_CLEANUP_START
+                               GET_AWAITABLE LOAD_CONST YIELD_FROM
+                               WITH_CLEANUP_FINISH END_FINALLY
+        async_with_stmt ::= expr
+                            BEFORE_ASYNC_WITH GET_AWAITABLE LOAD_CONST YIELD_FROM
+                            SETUP_ASYNC_WITH POP_TOP suite_stmts_opt
+                            POP_BLOCK LOAD_CONST
+                            COME_FROM_ASYNC_WITH
+                            WITH_CLEANUP_START
+                            GET_AWAITABLE LOAD_CONST YIELD_FROM
+                            WITH_CLEANUP_FINISH END_FINALLY
+
+        except_suite ::= c_stmts_opt COME_FROM POP_EXCEPT jump_except COME_FROM
+
+        # In 3.6+, A sequence of statements ending in a RETURN can cause
+        # JUMP_FORWARD END_FINALLY to be omitted from try middle
+
+        except_return ::= POP_TOP POP_TOP POP_TOP return_stmts
+        try_middle    ::= JUMP_FORWARD COME_FROM_EXCEPT except_return
+
+        # Try middle following a return_stmts
+        try_middle36    ::= COME_FROM_EXCEPT except_stmts END_FINALLY
+
+        stmt      ::= trystmt36
+        trystmt36 ::= SETUP_EXCEPT return_stmts try_middle36 opt_come_from_except
         """
 
     def add_custom_rules(self, tokens, customize):
         super(Python36Parser, self).add_custom_rules(tokens, customize)
         for i, token in enumerate(tokens):
-            opname = token.type
+            opname = token.kind
 
             if opname == 'FORMAT_VALUE':
                 rules_str = """
@@ -64,10 +101,10 @@ class Python36Parser(Python35Parser):
 
         if opname.startswith('CALL_FUNCTION_KW'):
             values = 'expr ' * token.attr
-            rule = 'call_function ::= expr kwargs_only_36 {token.type}'.format(**locals())
-            self.add_unique_rule(rule, token.type, token.attr, customize)
+            rule = 'call_function ::= expr kwargs_only_36 {token.kind}'.format(**locals())
+            self.add_unique_rule(rule, token.kind, token.attr, customize)
             rule = 'kwargs_only_36 ::= {values} LOAD_CONST'.format(**locals())
-            self.add_unique_rule(rule, token.type, token.attr, customize)
+            self.add_unique_rule(rule, token.kind, token.attr, customize)
         else:
             super(Python36Parser, self).custom_classfunc_rule(opname, token, customize)
 
@@ -78,10 +115,10 @@ class Python36ParserSingle(Python36Parser, PythonParserSingle):
 if __name__ == '__main__':
     # Check grammar
     p = Python36Parser()
-    p.checkGrammar()
+    p.check_grammar()
     from uncompyle6 import PYTHON_VERSION, IS_PYPY
     if PYTHON_VERSION == 3.6:
-        lhs, rhs, tokens, right_recursive = p.checkSets()
+        lhs, rhs, tokens, right_recursive = p.check_sets()
         from uncompyle6.scanner import get_scanner
         s = get_scanner(PYTHON_VERSION, IS_PYPY)
         opcode_set = set(s.opc.opname).union(set(

uncompyle6/parsers/parse37.py

@@ -0,0 +1,40 @@
+#  Copyright (c) 2017 Rocky Bernstein
+"""
+spark grammar differences over Python 3.6 for Python 3.7
+"""
+
+from uncompyle6.parser import PythonParserSingle
+from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
+from uncompyle6.parsers.parse36 import Python37Parser
+
+class Python36Parser(Python35Parser):
+
+    def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
+        super(Python37Parser, self).__init__(debug_parser)
+        self.customized = {}
+
+
+class Python37ParserSingle(Python37Parser, PythonParserSingle):
+    pass
+
+if __name__ == '__main__':
+    # Check grammar
+    p = Python37Parser()
+    p.check_grammar()
+    from uncompyle6 import PYTHON_VERSION, IS_PYPY
+    if PYTHON_VERSION == 3.7:
+        lhs, rhs, tokens, right_recursive = p.check_sets()
+        from uncompyle6.scanner import get_scanner
+        s = get_scanner(PYTHON_VERSION, IS_PYPY)
+        opcode_set = set(s.opc.opname).union(set(
+            """JUMP_BACK CONTINUE RETURN_END_IF COME_FROM
+               LOAD_GENEXPR LOAD_ASSERT LOAD_SETCOMP LOAD_DICTCOMP LOAD_CLASSNAME
+               LAMBDA_MARKER RETURN_LAST
+            """.split()))
+        remain_tokens = set(tokens) - opcode_set
+        import re
+        remain_tokens = set([re.sub('_\d+$', '', t) for t in remain_tokens])
+        remain_tokens = set([re.sub('_CONT$', '', t) for t in remain_tokens])
+        remain_tokens = set(remain_tokens) - opcode_set
+        print(remain_tokens)
+        # print(sorted(p.rule2name.items()))

uncompyle6/scanner.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016 by Rocky Bernstein
+#  Copyright (c) 2016-2017 by Rocky Bernstein
 #  Copyright (c) 2005 by Dan Pascu <dan@windowmaker.org>
 #  Copyright (c) 2000-2002 by hartmut Goebel <h.goebel@crazy-compilers.com>
 #  Copyright (c) 1999 John Aycock
@@ -10,17 +10,18 @@ scanner/ingestion module. From here we call various version-specific
 scanners, e.g. for Python 2.7 or 3.4.
 """
 
-from __future__ import print_function
-
 import sys
 
 from uncompyle6 import PYTHON3, IS_PYPY
 from uncompyle6.scanners.tok import Token
+from xdis.bytecode import op_size
+from xdis.magics import py_str2float
+from xdis.util import code2num
 
 # The byte code versions we support
 PYTHON_VERSIONS = (1.5,
                    2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7,
-                   3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 3.6)
+                   3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7)
 
 # FIXME: DRY
 if PYTHON3:
@@ -54,7 +55,7 @@ class Scanner(object):
 
         if version in PYTHON_VERSIONS:
             if is_pypy:
-                v_str = "opcode_pypy%s" % (int(version * 10))
+                v_str = "opcode_%spypy" % (int(version * 10))
             else:
                 v_str = "opcode_%s" % (int(version * 10))
             exec("from xdis.opcodes import %s" % v_str)
@@ -63,6 +64,7 @@ class Scanner(object):
             raise TypeError("%s is not a Python version I know about" % version)
 
         self.opname = self.opc.opname
+
         # FIXME: This weird Python2 behavior is not Python3
         self.resetTokenClass()
 
@@ -82,16 +84,21 @@ class Scanner(object):
             return True
         if self.code[offset] != self.opc.JUMP_ABSOLUTE:
             return False
-        return offset < self.get_target(offset)
+        # FIXME 0 isn't always correct
+        return offset < self.get_target(offset, 0)
 
     def get_target(self, pos, op=None):
         if op is None:
             op = self.code[pos]
         target = self.get_argument(pos)
-        if op in self.opc.hasjrel:
+        if op in self.opc.JREL_OPS:
             target += pos + 3
         return target
 
+    # FIXME: the below can be removed after xdis version 3.6.1 has been released
+    def extended_arg_val(self, val):
+        return val << self.opc.EXTENDED_ARG_SHIFT
+
     def get_argument(self, pos):
         arg = self.code[pos+1] + self.code[pos+2] * 256
         return arg
@@ -99,7 +106,7 @@ class Scanner(object):
     def print_bytecode(self):
         for i in self.op_range(0, len(self.code)):
             op = self.code[i]
-            if op in self.opc.hasjabs+self.opc.hasjrel:
+            if op in self.JUMP_OPS:
                 dest = self.get_target(i, op)
                 print('%i\t%s\t%i' % (i, self.opname[op], dest))
             else:
@@ -166,13 +173,20 @@ class Scanner(object):
 
         result_offset = None
         current_distance = len(code)
+        extended_arg = 0
         for offset in self.op_range(start, end):
             op = code[offset]
+
+            if op == self.opc.EXTENDED_ARG:
+                arg = code2num(code, offset+1) | extended_arg
+                extended_arg = self.extended_arg_val(arg)
+                continue
+
             if op in instr:
                 if target is None:
                     result_offset = offset
                 else:
-                    dest = self.get_target(offset)
+                    dest = self.get_target(offset, extended_arg)
                     if dest == target:
                         current_distance = 0
                         result_offset = offset
@@ -201,21 +215,32 @@ class Scanner(object):
             instr = [instr]
 
         result = []
+        extended_arg = 0
         for offset in self.op_range(start, end):
+
             op = code[offset]
+
+            if op == self.opc.EXTENDED_ARG:
+                arg = code2num(code, offset+1) | extended_arg
+                extended_arg = self.extended_arg_val(arg)
+                continue
+
             if op in instr:
                 if target is None:
                     result.append(offset)
                 else:
-                    t = self.get_target(offset)
+                    t = self.get_target(offset, extended_arg)
                     if include_beyond_target and t >= target:
                         result.append(offset)
                     elif t == target:
                         result.append(offset)
-        return result
+                        pass
+                    pass
+                pass
+            extended_arg = 0
+            pass
 
-    def op_hasArgument(self, op):
-        return self.op_size(op) > 1
+        return result
 
     def op_range(self, start, end):
         """
@@ -224,20 +249,7 @@ class Scanner(object):
         """
         while start < end:
             yield start
-            start += self.op_size(self.code[start])
-
-    def next_offset(self, op, offset):
-        return offset + self.op_size(op)
-
-    def op_size(self, op):
-        """
-        Return size of operator with its arguments
-        for given opcode <op>.
-        """
-        if op < self.opc.HAVE_ARGUMENT:
-            return 2 if self.version >= 3.6 else 1
-        else:
-            return 2 if self.version >= 3.6 else 3
+            start += op_size(self.code[start], self.opc)
 
     def remove_mid_line_ifs(self, ifs):
         """
@@ -269,13 +281,16 @@ class Scanner(object):
         self.Token = tokenClass
         return self.Token
 
-def op_has_argument(op, opc):
-    return op >= opc.HAVE_ARGUMENT
-
 def parse_fn_counts(argc):
     return ((argc & 0xFF), (argc >> 8) & 0xFF, (argc >> 16) & 0x7FFF)
 
+
 def get_scanner(version, is_pypy=False, show_asm=None):
+
+    # If version is a string, turn that into the corresponding float.
+    if isinstance(version, str):
+        version = py_str2float(version)
+
     # Pick up appropriate scanner
     if version in PYTHON_VERSIONS:
         v_str = "%s" % (int(version * 10))
@@ -302,5 +317,7 @@ def get_scanner(version, is_pypy=False, show_asm=None):
 if __name__ == "__main__":
     import inspect, uncompyle6
     co = inspect.currentframe().f_code
+    scanner = get_scanner('2.7.13', True)
+    scanner = get_scanner(sys.version[:5], False)
     scanner = get_scanner(uncompyle6.PYTHON_VERSION, IS_PYPY, True)
     tokens, customize = scanner.ingest(co, {})

uncompyle6/scanners/pypy27.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016 by Rocky Bernstein
+#  Copyright (c) 2016-2017 by Rocky Bernstein
 """
 Python PyPy 2.7 bytecode scanner/deparser
 
@@ -10,8 +10,8 @@ information for later use in deparsing.
 import uncompyle6.scanners.scanner27 as scan
 
 # bytecode verification, verify(), uses JUMP_OPs from here
-from xdis.opcodes import opcode_pypy27
-JUMP_OPs = opcode_pypy27.JUMP_OPs
+from xdis.opcodes import opcode_27pypy
+JUMP_OPS = opcode_27pypy.JUMP_OPS
 
 # We base this off of 2.6 instead of the other way around
 # because we cleaned things up this way.

uncompyle6/scanners/pypy32.py

@@ -1,22 +1,18 @@
-#  Copyright (c) 2016 by Rocky Bernstein
+#  Copyright (c) 2017 by Rocky Bernstein
 """
-Python PyPy 3.2 bytecode scanner/deparser
+Python PyPy 3.2 decompiler scanner.
 
-This overlaps Python's 3.2's dis module, but it can be run from
-Python 3 and other versions of Python. Also, we save token
-information for later use in deparsing.
+Does some additional massaging of xdis-disassembled instructions to
+make things easier for decompilation.
 """
 
 import uncompyle6.scanners.scanner32 as scan
 
 # bytecode verification, verify(), uses JUMP_OPs from here
-from xdis.opcodes import opcode_32 as opc  # is this rgith?
+from xdis.opcodes import opcode_32 as opc  # is this right?
 JUMP_OPs = map(lambda op: opc.opname[op], opc.hasjrel + opc.hasjabs)
 
-# We base this off of 2.6 instead of the other way around
-# because we cleaned things up this way.
-# The history is that 2.7 support is the cleanest,
-# then from that we got 2.6 and so on.
+# We base this off of 3.2
 class ScannerPyPy32(scan.Scanner32):
     def __init__(self, show_asm):
         # There are no differences in initialization between

uncompyle6/scanners/pypy35.py

@@ -0,0 +1,22 @@
+#  Copyright (c) 2017 by Rocky Bernstein
+"""
+Python PyPy 3.2 decompiler scanner.
+
+Does some additional massaging of xdis-disassembled instructions to
+make things easier for decompilation.
+"""
+
+import uncompyle6.scanners.scanner35 as scan
+
+# bytecode verification, verify(), uses JUMP_OPS from here
+from xdis.opcodes import opcode_35 as opc  # is this right?
+JUMP_OPs = opc.JUMP_OPS
+
+# We base this off of 3.5
+class ScannerPyPy35(scan.Scanner35):
+    def __init__(self, show_asm):
+        # There are no differences in initialization between
+        # pypy 3.5 and 3.5
+        scan.Scanner35.__init__(self, show_asm, is_pypy=True)
+        self.version = 3.5
+        return

uncompyle6/scanners/scanner15.py

@@ -1,6 +1,6 @@
-#  Copyright (c) 2016 by Rocky Bernstein
+#  Copyright (c) 2016-2017 by Rocky Bernstein
 """
-Python 1.5 bytecode scanner/deparser
+Python 1.5 bytecode decompiler scanner.
 
 This massages tokenized 1.5 bytecode to make it more amenable for
 grammar parsing.
@@ -11,15 +11,15 @@ import uncompyle6.scanners.scanner21 as scan
 
 # bytecode verification, verify(), uses JUMP_OPs from here
 from xdis.opcodes import opcode_15
-JUMP_OPs = opcode_15.JUMP_OPs
+JUMP_OPS = opcode_15.JUMP_OPS
 
-# We base this off of 2.2 instead of the other way around
+# We base this off of 2.1 instead of the other way around
 # because we cleaned things up this way.
 # The history is that 2.7 support is the cleanest,
 # then from that we got 2.6 and so on.
 class Scanner15(scan.Scanner21):
     def __init__(self, show_asm=False):
-        scan.Scanner21.__init__(self, show_asm)
+        scan.Scanner21.__init__(self, show_asm=False)
         self.opc = opcode_15
         self.opname = opcode_15.opname
         self.version = 1.5

uncompyle6/scanners/scanner2.py

@@ -1,4 +1,4 @@
-# Copyright (c) 2015, 2016 by Rocky Bernstein
+# Copyright (c) 2015-2017 by Rocky Bernstein
 # Copyright (c) 2005 by Dan Pascu <dan@windowmaker.org>
 # Copyright (c) 2000-2002 by hartmut Goebel <h.goebel@crazy-compilers.com>
 """
@@ -20,19 +20,24 @@ For example:
 Finally we save token information.
 """
 
-from __future__ import print_function
+from uncompyle6 import PYTHON_VERSION
+
+if PYTHON_VERSION < 2.6:
+    from xdis.namedtuple24 import namedtuple
+else:
+    from collections import namedtuple
 
-from collections import namedtuple
 from array import array
 
-from uncompyle6.scanner import op_has_argument
 from xdis.code import iscode
+from xdis.bytecode import op_has_argument, op_size, instruction_size
+from xdis.util import code2num
 
-import uncompyle6.scanner as scan
+from uncompyle6.scanner import Scanner
 
-class Scanner2(scan.Scanner):
+class Scanner2(Scanner):
     def __init__(self, version, show_asm=None, is_pypy=False):
-        scan.Scanner.__init__(self, version, show_asm, is_pypy)
+        Scanner.__init__(self, version, show_asm, is_pypy)
         self.pop_jump_if = frozenset([self.opc.PJIF, self.opc.PJIT])
         self.jump_forward = frozenset([self.opc.JUMP_ABSOLUTE, self.opc.JUMP_FORWARD])
         # This is the 2.5+ default
@@ -84,20 +89,28 @@ class Scanner2(scan.Scanner):
         cause specific rules for the specific number of arguments they take.
         """
 
-        show_asm = self.show_asm if not show_asm else show_asm
+        if not show_asm:
+            show_asm = self.show_asm
+
         # show_asm = 'after'
         if show_asm in ('both', 'before'):
             from xdis.bytecode import Bytecode
             bytecode = Bytecode(co, self.opc)
             for instr in bytecode.get_instructions(co):
-                print(instr._disassemble())
+                print(instr.disassemble())
 
-        # Container for tokens
+        # list of tokens/instructions
         tokens = []
 
+        # "customize" is a dict whose keys are nonterminals
+        # and the value is the argument stack entries for that
+        # nonterminal. The count is a little hoaky. It is mostly
+        # not used, but sometimes it is.
+        # "customize" is a dict whose keys are nonterminals
         customize = {}
+
         if self.is_pypy:
-            customize['PyPy'] = 1
+            customize['PyPy'] = 0
 
         Token = self.Token # shortcut
 
@@ -186,9 +199,9 @@ class Scanner2(scan.Scanner):
                 oparg = self.get_argument(offset) + extended_arg
                 extended_arg = 0
                 if op == self.opc.EXTENDED_ARG:
-                    extended_arg = oparg * scan.L65536
+                    extended_arg += self.extended_arg_val(oparg)
                     continue
-                if op in self.opc.hasconst:
+                if op in self.opc.CONST_OPS:
                     const = co.co_consts[oparg]
                     if iscode(const):
                         oparg = const
@@ -209,23 +222,23 @@ class Scanner2(scan.Scanner):
                         pattr = '<code_object ' + const.co_name + '>'
                     else:
                         pattr = const
-                elif op in self.opc.hasname:
+                elif op in self.opc.NAME_OPS:
                     pattr = names[oparg]
-                elif op in self.opc.hasjrel:
+                elif op in self.opc.JREL_OPS:
                     #  use instead: hasattr(self, 'patch_continue'): ?
                     if self.version == 2.7:
                         self.patch_continue(tokens, offset, op)
                     pattr = repr(offset + 3 + oparg)
-                elif op in self.opc.hasjabs:
+                elif op in self.opc.JABS_OPS:
                     # use instead: hasattr(self, 'patch_continue'): ?
                     if self.version == 2.7:
                         self.patch_continue(tokens, offset, op)
                     pattr = repr(oparg)
-                elif op in self.opc.haslocal:
+                elif op in self.opc.LOCAL_OPS:
                     pattr = varnames[oparg]
-                elif op in self.opc.hascompare:
+                elif op in self.opc.COMPARE_OPS:
                     pattr = self.opc.cmp_op[oparg]
-                elif op in self.opc.hasfree:
+                elif op in self.opc.FREE_OPS:
                     pattr = free[oparg]
 
             if op in self.varargs_ops:
@@ -327,7 +340,7 @@ class Scanner2(scan.Scanner):
         for i in self.op_range(0, n):
             op = self.code[i]
             self.prev.append(i)
-            if self.op_hasArgument(op):
+            if op_has_argument(op, self.opc):
                 self.prev.append(i)
                 self.prev.append(i)
                 pass
@@ -380,7 +393,7 @@ class Scanner2(scan.Scanner):
                     if elem != code[i]:
                         match = False
                         break
-                    i += self.op_size(code[i])
+                    i += op_size(code[i], self.opc)
 
                 if match:
                     i = self.prev[i]
@@ -451,7 +464,7 @@ class Scanner2(scan.Scanner):
                     self.not_continue.add(jmp)
                     jmp = self.get_target(jmp)
                     prev_offset = self.prev[except_match]
-                    # COMPARE_OP argument should be "exception match" or 10
+                    # COMPARE_OP argument should be "exception-match" or 10
                     if (self.code[prev_offset] == self.opc.COMPARE_OP and
                         self.code[prev_offset+1] != 10):
                         return None
@@ -478,7 +491,7 @@ class Scanner2(scan.Scanner):
             elif op in self.setup_ops:
                 count_SETUP_ += 1
 
-    def detect_control_flow(self, offset, op):
+    def detect_control_flow(self, offset, op, extended_arg):
         """
         Detect type of block structures and their boundaries to fix optimized jumps
         in python2.3+
@@ -502,14 +515,13 @@ class Scanner2(scan.Scanner):
                 parent = struct
 
         if op == self.opc.SETUP_LOOP:
-
             # We categorize loop types: 'for', 'while', 'while 1' with
             # possibly suffixes '-loop' and '-else'
             # Try to find the jump_back instruction of the loop.
             # It could be a return instruction.
 
-            start = offset+3
-            target = self.get_target(offset, op)
+            start += instruction_size(op, self.opc)
+            target = self.get_target(offset) + extended_arg
             end    = self.restrict_to_parent(target, parent)
             self.setup_loop_targets[offset] = target
             self.setup_loops[target] = offset
@@ -602,7 +614,7 @@ class Scanner2(scan.Scanner):
 
                     if test == offset:
                         loop_type = 'while 1'
-                    elif self.code[test] in self.opc.hasjabs + self.opc.hasjrel:
+                    elif self.code[test] in self.opc.JUMP_OPs:
                         self.ignore_if.add(test)
                         test_target = self.get_target(test)
                         if test_target > (jump_back+3):
@@ -617,7 +629,7 @@ class Scanner2(scan.Scanner):
                                        'start': jump_back+3,
                                        'end':   end})
         elif op == self.opc.SETUP_EXCEPT:
-            start  = offset + self.op_size(op)
+            start  = offset + op_size(op, self.opc)
             target = self.get_target(offset, op)
             end    = self.restrict_to_parent(target, parent)
             if target != end:
@@ -641,7 +653,7 @@ class Scanner2(scan.Scanner):
                         setup_except_nest -= 1
                 elif self.code[end_finally_offset] == self.opc.SETUP_EXCEPT:
                     setup_except_nest += 1
-                end_finally_offset += self.op_size(code[end_finally_offset])
+                end_finally_offset += op_size(code[end_finally_offset], self.opc)
                 pass
 
             # Add the except blocks
@@ -842,7 +854,7 @@ class Scanner2(scan.Scanner):
                     else:
                         # We still have the case in 2.7 that the next instruction
                         # is a jump to a SETUP_LOOP target.
-                        next_offset = target + self.op_size(self.code[target])
+                        next_offset = target + op_size(self.code[target], self.opc)
                         next_op = self.code[next_offset]
                         if self.op_name(next_op) == 'JUMP_FORWARD':
                             jump_target = self.get_target(next_offset, next_op)
@@ -904,7 +916,9 @@ class Scanner2(scan.Scanner):
                                          'start': start-3,
                                          'end':   pre_rtarget})
 
-                self.not_continue.add(pre_rtarget)
+                # FIXME: this is yet another case were we need dominators.
+                if pre_rtarget not in self.linestartoffsets or self.version < 2.7:
+                    self.not_continue.add(pre_rtarget)
 
                 if rtarget < end:
                     # We have an "else" block  of some kind.
@@ -979,23 +993,29 @@ class Scanner2(scan.Scanner):
         self.thens = {} # JUMP_IF's that separate the 'then' part of an 'if'
 
         targets = {}
+        extended_arg = 0
         for offset in self.op_range(0, n):
             op = code[offset]
 
+            if op == self.opc.EXTENDED_ARG:
+                arg = code2num(code, offset+1) | extended_arg
+                extended_arg += self.extended_arg_val(arg)
+                continue
+
             # Determine structures and fix jumps in Python versions
             # since 2.3
-            self.detect_control_flow(offset, op)
+            self.detect_control_flow(offset, op, extended_arg)
 
             if op_has_argument(op, self.opc):
                 label = self.fixed_jumps.get(offset)
                 oparg = self.get_argument(offset)
 
                 if label is None:
-                    if op in self.opc.hasjrel and self.op_name(op) != 'FOR_ITER':
-                        # if (op in self.opc.hasjrel and
+                    if op in self.opc.JREL_OPS and self.op_name(op) != 'FOR_ITER':
+                        # if (op in self.opc.JREL_OPS and
                         #     (self.version < 2.0 or op != self.opc.FOR_ITER)):
                         label = offset + 3 + oparg
-                    elif self.version == 2.7 and op in self.opc.hasjabs:
+                    elif self.version == 2.7 and op in self.opc.JABS_OPS:
                         if op in (self.opc.JUMP_IF_FALSE_OR_POP,
                                   self.opc.JUMP_IF_TRUE_OR_POP):
                             if (oparg > offset):
@@ -1023,8 +1043,10 @@ class Scanner2(scan.Scanner):
 
                             # FIXME: rocky: I think we need something like this...
                             if offset not in set(self.ignore_if) or self.version == 2.7:
-                                source = (self.setup_loops[label]
-                                          if label in self.setup_loops else offset)
+                                if label in self.setup_loops:
+                                    source = self.setup_loops[label]
+                                else:
+                                    source = offset
                                 targets[label] = targets.get(label, []) + [source]
                             pass
 
@@ -1034,7 +1056,9 @@ class Scanner2(scan.Scanner):
                 label = self.fixed_jumps[offset]
                 targets[label] = targets.get(label, []) + [offset]
                 pass
-            pass
+
+            extended_arg = 0
+            pass # for loop
 
         # DEBUG:
         if debug in ('both', 'after'):

uncompyle6/scanners/scanner21.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016 by Rocky Bernstein
+#  Copyright (c) 2016-2017 by Rocky Bernstein
 """
 Python 2.1 bytecode scanner/deparser
 
@@ -11,7 +11,7 @@ import uncompyle6.scanners.scanner22 as scan
 
 # bytecode verification, verify(), uses JUMP_OPs from here
 from xdis.opcodes import opcode_21
-JUMP_OPs = opcode_21.JUMP_OPs
+JUMP_OPS = opcode_21.JUMP_OPS
 
 # We base this off of 2.2 instead of the other way around
 # because we cleaned things up this way.
@@ -19,7 +19,7 @@ JUMP_OPs = opcode_21.JUMP_OPs
 # then from that we got 2.6 and so on.
 class Scanner21(scan.Scanner22):
     def __init__(self, show_asm=False):
-        scan.Scanner22.__init__(self, show_asm)
+        scan.Scanner22.__init__(self, show_asm=False)
         self.opc = opcode_21
         self.opname = opcode_21.opname
         self.version = 2.1

uncompyle6/scanners/scanner22.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016 by Rocky Bernstein
+#  Copyright (c) 2016-2017 by Rocky Bernstein
 """
 Python 2.2 bytecode ingester.
 
@@ -11,7 +11,7 @@ import uncompyle6.scanners.scanner23 as scan
 
 # bytecode verification, verify(), uses JUMP_OPs from here
 from xdis.opcodes import opcode_22
-JUMP_OPs = opcode_22.JUMP_OPs
+JUMP_OPS = opcode_22.JUMP_OPS
 
 # We base this off of 2.3 instead of the other way around
 # because we cleaned things up this way.
@@ -19,7 +19,7 @@ JUMP_OPs = opcode_22.JUMP_OPs
 # then from that we got 2.6 and so on.
 class Scanner22(scan.Scanner23):
     def __init__(self, show_asm=False):
-        scan.Scanner23.__init__(self, show_asm)
+        scan.Scanner23.__init__(self, show_asm=False)
         self.opc = opcode_22
         self.opname = opcode_22.opname
         self.version = 2.2
@@ -30,5 +30,5 @@ class Scanner22(scan.Scanner23):
 
     def ingest22(self, co, classname=None, code_objects={}, show_asm=None):
         tokens, customize = self.parent_ingest(co, classname, code_objects, show_asm)
-        tokens = [t for t in tokens if t.type != 'SET_LINENO']
+        tokens = [t for t in tokens if t.kind != 'SET_LINENO']
         return tokens, customize

uncompyle6/scanners/scanner23.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016 by Rocky Bernstein
+#  Copyright (c) 2016-2017 by Rocky Bernstein
 """
 Python 2.3 bytecode scanner/deparser
 
@@ -10,7 +10,7 @@ import uncompyle6.scanners.scanner24 as scan
 
 # bytecode verification, verify(), uses JUMP_OPs from here
 from xdis.opcodes import opcode_23
-JUMP_OPs = opcode_23.JUMP_OPs
+JUMP_OPS = opcode_23.JUMP_OPS
 
 # We base this off of 2.4 instead of the other way around
 # because we cleaned things up this way.

uncompyle6/scanners/scanner24.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016 by Rocky Bernstein
+#  Copyright (c) 2016-2017 by Rocky Bernstein
 """
 Python 2.4 bytecode scanner/deparser
 
@@ -10,7 +10,7 @@ import uncompyle6.scanners.scanner25 as scan
 
 # bytecode verification, verify(), uses JUMP_OPs from here
 from xdis.opcodes import opcode_24
-JUMP_OPs = opcode_24.JUMP_OPs
+JUMP_OPS = opcode_24.JUMP_OPS
 
 # We base this off of 2.5 instead of the other way around
 # because we cleaned things up this way.

uncompyle6/scanners/scanner25.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2015-2016 by Rocky Bernstein
+#  Copyright (c) 2015-2017 by Rocky Bernstein
 """
 Python 2.5 bytecode scanner/deparser
 
@@ -11,7 +11,7 @@ import uncompyle6.scanners.scanner26 as scan
 
 # bytecode verification, verify(), uses JUMP_OPs from here
 from xdis.opcodes import opcode_25
-JUMP_OPs = opcode_25.JUMP_OPs
+JUMP_OPS = opcode_25.JUMP_OPS
 
 # We base this off of 2.6 instead of the other way around
 # because we cleaned things up this way.