2.4-branch uncompyle6 doesn't support -c

More is needed though.

.travis.yml

@@ -1,10 +1,13 @@
 language: python
 
-sudo: false
-
 python:
   - '2.7' # this is a cheat here because travis doesn't do 2.4-2.6
 
+matrix:
+  include:
+    - python: '2.7'
+      dist: xenial  # required for Python >= 3.7 (travis-ci/travis-ci#9069)
+
 install:
 - pip install -e .
 - pip install -r requirements-dev.txt

HOW-TO-REPORT-A-BUG.md

@@ -1,6 +1,21 @@
-# How to report a Bug
+<!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc -->
+**Table of Contents**
 
-## The difficulty of the problem
+- [The difficulty of the problem](#the-difficulty-of-the-problem)
+- [Is it really a bug?](#is-it-really-a-bug)
+    - [Do you have valid bytecode?](#do-you-have-valid-bytecode)
+    - [Semantic equivalence vs. exact source code](#semantic-equivalence-vs-exact-source-code)
+- [What to send (minimum requirements)](#what-to-send-minimum-requirements)
+- [What to send (additional helpful information)](#what-to-send-additional-helpful-information)
+    - [But I don't *have* the source code!](#but-i-dont-have-the-source-code)
+    - [But I don't *have* the source code and am incapable of figuring how how to do a hand disassembly!](#but-i-dont-have-the-source-code-and-am-incapable-of-figuring-how-how-to-do-a-hand-disassembly)
+- [Narrowing the problem](#narrowing-the-problem)
+- [Karma](#karma)
+- [Confidentiality of Bug Reports](#confidentiality-of-bug-reports)
+- [Ethics](#ethics)
+
+<!-- markdown-toc end -->
+# The difficulty of the problem
 
 This decompiler is a constant work in progress: Python keeps
 changing, and so does its code generation.
@@ -41,10 +56,10 @@ bugs you may have an interest in. If you require decompiling bytecode
 immediately, consider using a decompilation service, listed further
 down in this document.
 
-## Is it really a bug?
+# Is it really a bug?
 
 
-### Do you have valid bytecode?
+## Do you have valid bytecode?
 
 As mentioned in README.rst, this project doesn't handle obfuscated
 code. See README.rst for suggestions for how to remove some kinds of
@@ -55,11 +70,11 @@ Python comes with a disassembly module called `dis`. A prerequisite
 module for this package, `xdis` has a cross-python version
 disassembler called `pydisasm`.
 
-### Semantic equivalence vs. exact source code
+## Semantic equivalence vs. exact source code
 
 Consider how Python compiles something like "(x*y) + 5". Early on
 Python creates an "abstract syntax tree" (AST) for this. And this is
-"abstract" in the sense that unimportant, redundant or unnecceary
+"abstract" in the sense that unimportant, redundant or unnecessary
 items have been removed. Here, this means that any notion that you
 wrote "x+y" in parenthesis is lost, since in this context they are
 unneeded. Also lost is the fact that the multiplication didn't have
@@ -132,7 +147,7 @@ Python will eliminate the entire "if" statement.
 So just because the text isn't the same, does not
 necessarily mean there's a bug.
 
-## What to send (minimum requirements)
+# What to send (minimum requirements)
 
 The basic requirement is pretty simple:
 
@@ -146,7 +161,7 @@ sending is too large.
 
 Also try to narrow the bug. See below.
 
-## What to send (additional helpful information)
+# What to send (additional helpful information)
 
 Some kind folks also give the invocation they used and the output
 which usually includes an error message produced. This is
@@ -159,7 +174,7 @@ provide the input command and the output from that, please give:
 * Python interpreter version used
 
 
-### But I don't *have* the source code!
+## But I don't *have* the source code!
 
 Sure, I get it. No problem. There is Python assembly code on parse
 errors, so simply by hand decompile that. To get a full disassembly,
@@ -167,7 +182,7 @@ use `pydisasm` from the [xdis](https://pypi.python.org/pypi/xdis)
 package. Opcodes are described in the documentation for
 the [dis](https://docs.python.org/3.6/library/dis.html) module.
 
-### But I don't *have* the source code and am incapable of figuring how how to do a hand disassembly!
+### But I don't *have* the source code and am incapable of figuring how to do a hand disassembly!
 
 Well, you could learn. No one is born into this world knowing how to
 disassemble Python bytecode. And as Richard Feynman once said, "What
@@ -179,7 +194,7 @@ Compilers](http://www.crazy-compilers.com/decompyle/) offers a
 byte-code decompiler service for versions of Python up to 2.6. (If
 there are others around let me know and I'll list them here.)
 
-## Narrowing the problem
+# Narrowing the problem
 
 I don't need or want the entire source code base for the file(s) or
 module(s) can't be decompiled. I just need those file(s) or module(s).
@@ -197,22 +212,53 @@ what doesn't. That is useful. Or maybe the same file will decompile
 properly on a neighboring version of Python. That is helpful too.
 
 In sum, the more you can isolate or narrow the problem, the more
-likley the problem will be fixed and fixed sooner.
+likely the problem will be fixed and fixed sooner.
 
-## Confidentiality of Bug Reports
+# Karma
+
+I realize that following the instructions given herein puts a bit of
+burden on the bug reporter. In my opinion, this is justified as
+attempts to balance somewhat the burden and effort needed to fix the
+bug and the attempts to balance number of would-be bug reporters with
+the number of bug fixers. Better bug reporters are more likely to move
+in the category of bug fixers.
+
+The barrier to reporting a big is pretty small: all you really need is
+a github account, and the ability to type something after clicking
+some buttons. So the reality is that many people just don't bother to
+read these instructions, let alone follow it to any simulacrum.
+
+And the reality is also that bugs sometimes get fixed even though
+these instructions are not followed.
+
+So one factors I may take into consideration is the bug reporter's karma.
+
+* Have you demonstrably contributed to open source? I may look at your
+  github profile to see what contributions you have made, how popular
+  those contributions are, or how popular you are.
+* How appreciative are you? Have you starred this project that you are
+  seeking help from? Have you starred _any_ github project? And the above
+  two kind of feed into ...
+* Attitude. Some people feel that they are doing me and the world a
+  great favor by just pointing out that there is a problem whose solution
+  would greatly benefit them. Perhaps this is why they feel that
+  instructions are not to be followed by them, nor any need for
+  showing evidence gratitude when help is offered them.
+
+# Confidentiality of Bug Reports
 
 When you report a bug, you are giving up confidentiality to the source
 code and the byte code. However, I would imagine that if you have
 narrowed the problem sufficiently, confidentiality of the little that
 remains would not be an issue.
 
-However feel free to remove any commments, and modify variable names
+However feel free to remove any comments, and modify variable names
 or constants in the source code.
 
-## Ethics
+# Ethics
 
 I do not condone using this program for unethical or illegal purposes.
-More detestful, at least to me, is asking for help to assist you in
+More detestable, at least to me, is asking for help to assist you in
 something that might not legitimate.
 
 Don't use the issue tracker for such solicitations. To try to stave

NEWS.md

@@ -1,36 +1,80 @@
-uncompyle6 3.2.4 2018-06-04 7x9 release
+3.3.0 2019-03-23 Holy Week
+==========================
+
+* First cut at Python 3.8 (many bug remain)
+* The usual smattering of bug and doc fixes
+
+3.2.6 2019-03-23 Mueller Report
+=======================================
+
+Mostly more of the same: bug fixes and pull requests.
+
+Bug Fixes
+-----------
+
+* [#155: Python 3.x bytecode confusing "try/else" with "try" in a loop](https://github.com/rocky/python-uncompyle6/issues/155),
+* [#200: Python 3 bug in not detecting end bounds of an "if" ... "elif"](https://github.com/rocky/python-uncompyle6/issues/200),
+* [#208: Comma placement in 3.6 and 3.7 **kwargs](https://github.com/rocky/python-uncompyle6/issues/208),
+* [#209: Fix "if" return boundary in 3.6+](https://github.com/rocky/python-uncompyle6/issues/209),
+* [#215: 2.7 can have two JUMP_BACKs at the end of a while loop](https://github.com/rocky/python-uncompyle6/issues/215)
+
+Pull Requests
+----------------
+
+* [#202: Better "assert" statement detemination in Python 2.7](https://github.com/rocky/python-uncompyle6/pull/211)
+* [#204: Python 3.7 testing](https://github.com/rocky/python-uncompyle6/pull/204)
+* [#205: Run more f-string tests on Python 3.7](https://github.com/rocky/python-uncompyle6/pull/205)
+* [#211: support utf-8 chars in Python 3 sourcecode](https://github.com/rocky/python-uncompyle6/pull/202)
+
+
+
+3.2.5 2018-12-30 Clearout sale
+======================================
+
+- 3.7.2 Remove deprecation warning on regexp string that isn't raw
+- main.main() parameter `codes` is not used - note that
+- Improve Python 3.6+ control flow detection
+- More complete fragment instruction annotation for `imports`
+
+3.2.4 2018-10-27 7x9 release
+===================================
 
 - Bug fixes #180, #182, #187, #192
 - Enhancements #189
 - Internal improvements
 
-uncompyle6 3.2.3 2018-06-04 Michael Cohen flips and Fleetwood Redux
-
+3.2.3 2018-06-04 Michael Cohen flips and Fleetwood Redux
+======================================================================
 - Python 1.3 support 3.0 bug and
 - fix botched parameter ordering of 3.x in last release
 
-uncompyle6 3.2.2 2018-06-04 When I'm 64
+3.2.2 2018-06-04 When I'm 64
+===================================
 
 - Python 3.0 support and bug fixes
 
-uncompyle6 3.2.1 2018-06-04 MF
+3.2.1 2018-06-04 MF
+=======================
 
 - Python 1.4 and 1.5 bug fixes
 
-uncompyle6 3.2.0 2018-05-19 Rocket Scientist
+3.2.0 2018-05-19 Rocket Scientist
+=========================================
 
 - Add rudimentary 1.4 support (still a bit buggy)
 - add --tree+ option to show formatting rule, when it is constant
 - Python 2.7.15candidate1 support (via xdis)
 - bug fixes, especially for 3.7 (but 2.7 and 3.6 and others as well)
 
-uncompyle6 3.1.3 2018-04-16
+3.1.3 2018-04-16
+====================
 
 - Add some Python 3.7 rules, such as for handling LOAD_METHOD (not complete)
 - Fix some fragment bugs
 - small doc changes
 
-uncompyle6 3.1.2 2018-04-08 Eastern Orthodox Easter
+3.1.2 2018-04-08 Eastern Orthodox Easter
+==================================================
 
 - Python 3.x subclass and call parsing fixes
 - Allow/note running on Python 3.1
@@ -38,7 +82,8 @@ uncompyle6 3.1.2 2018-04-08 Eastern Orthodox Easter
 - DRY instruction building code between 2.x and 3.x
 - expand testing
 
-uncompyle6 3.1.1 2018-04-01 Easter April Fool's
+3.1.1 2018-04-01 Easter April Fool's
+=============================================
 
 Jesus on Friday's New York Times puzzle: "I'm stuck on 2A"
 
@@ -51,7 +96,8 @@ Jesus on Friday's New York Times puzzle: "I'm stuck on 2A"
 - more runtime testing of decompiled code
 - more removal of parenthesis around calls via setting precidence
 
-uncompyle6 3.1.0 2018-03-21 Equinox
+3.1.0 2018-03-21 Equinox
+==============================
 
 - Add code_deparse_with_offset() fragment function.
 - Correct paramenter call fragment deparse_code()
@@ -59,7 +105,8 @@ uncompyle6 3.1.0 2018-03-21 Equinox
   About 5% of 3.6 fail parsing now. But
   semantics still needs much to be desired.
 
-uncompyle6 3.0.1 2018-02-17
+3.0.1 2018-02-17
+====================
 
 - All Python 2.6.9 standard library files weakly verify
 - Many 3.6 fixes. 84% of the first 200 standard library files weakly compile.
@@ -69,7 +116,8 @@ uncompyle6 3.0.1 2018-02-17
 - And more add tests target previous existing bugs more completely
 - sync recent license changes in metadata
 
-uncompyle6 3.0.0 2018-02-17
+3.0.0 2018-02-17
+====================
 
 - deparse_code() and lookalikes from the various semantic actions are
   now deprecated. Instead use new API code_deparse() which makes the
@@ -91,7 +139,8 @@ A bit more work is still needed for 3.6 especially in the area of
 function calls and definitions.
 
 
-uncompyle6 2.16.0 2018-02-17
+2.16.0 2018-02-17
+=====================
 
 - API additions:
   - add fragments.op_at_code_loc() and
@@ -103,18 +152,21 @@ uncompyle6 2.16.0 2018-02-17
 - Fix Python 3.5+ CALL_FUNCTION_VAR and BUILD_LIST_UNPACK in call; with this
   we can can handle 3.5+ f(a, b, *c, *d, *e) now
 
-uncompyle6 2.15.1 2018-02-05
+2.15.1 2018-02-05
+=====================
 
 - More bug fixes and revert an improper bug fix in 2.15.0
 
-uncompyle6 2.15.0 2018-02-05 pycon2018.co
+2.15.0 2018-02-05 pycon2018.co
+=====================================
 
 - Bug fixes
 - Code fragment improvements
 - Code cleanups
 - Expand testing
 
-uncompyle6 2.15.1 2018-01-27
+2.15.1 2018-01-27
+=====================
 
 - Add --linemap option to give line correspondences
   between original source lines and reconstructed line sources.
@@ -126,7 +178,8 @@ uncompyle6 2.15.1 2018-01-27
 - Correct 3.6+ calls with kwargs
 - Describe the difficulty of 3.6 in README
 
-uncompyle6 2.14.3 2018-01-19
+2.14.3 2018-01-19
+=====================
 
 - Fix bug in 3.5+ await stmt
 - Better version to magic handling; handle 3.5.2 .. 3.5.4 versions
@@ -138,7 +191,8 @@ uncompyle6 2.14.3 2018-01-19
 - better tests in setup.py for running the right version of Python
 - Fix 2.6- parsing of "for .. try/else" ...  with "continue"  inside
 
-uncompyle6 2.14.2 2018-01-09 Samish
+2.14.2 2018-01-09 Samish
+==============================
 
 Decompilation bug fixes, mostly 3.6 and pre 2.7
 
@@ -156,7 +210,8 @@ Decompilation bug fixes, mostly 3.6 and pre 2.7
   Python versions
 - Match Python AST names more closely when possible
 
-uncompyle6 2.14.1 2017-12-10 Dr. Gecko
+2.14.1 2017-12-10 Dr. Gecko
+===================================
 
 - Many decompilation bugfixes
 - Grammar rule reduction and version isolation
@@ -164,7 +219,8 @@ uncompyle6 2.14.1 2017-12-10 Dr. Gecko
   with Python AST
 - Start automated Python stdlib testing - full round trip
 
-uncompyle6 2.14.0 2017-11-26 johnnybamazing
+2.14.0 2017-11-26 johnnybamazing
+=========================================
 
 - Start to isolate grammar rules between versions
   and remove used grammar rules
@@ -172,7 +228,8 @@ uncompyle6 2.14.0 2017-11-26 johnnybamazing
   (many more remain)
 - Add stdlib/runtests.sh for even more rigorous testing
 
-uncompyle6 2.13.3 2017-11-13
+2.13.3 2017-11-13
+=====================
 
 Overall: better 3.6 decompiling and some much needed code refactoring and cleanup
 
@@ -198,22 +255,26 @@ Overall: better 3.6 decompiling and some much needed code refactoring and cleanu
 - reinstate some bytecode tests since decompiling has gotten better
 - Revise how to report a bug
 
-uncompyle6 2.13.2 2017-10-12
+2.13.2 2017-10-12
+=====================
 
 - Re-release using a more automated approach
 
-uncompyle6 2.13.1 2017-10-11
+2.13.1 2017-10-11
+=====================
 
 - Re-release because Python 2.4 source uploaded rather than 2.6-3.6
 
-uncompyle6 2.13.0 2017-10-10
+2.13.0 2017-10-10
+=====================
 
 - Fixes in deparsing lambda expressions
 - Improve table-semantics descriptions
 - Document hacky customize arg count better (until we can remove it)
 - Update to use xdis 3.7.0 or greater
 
-uncompyle6 2.12.0 2017-09-26
+2.12.0 2017-09-26
+=====================
 
 - Use xdis 3.6.0 or greater now
 - Small semantic table cleanups
@@ -221,11 +282,13 @@ uncompyle6 2.12.0 2017-09-26
 - Slightly more Python 3.7, but still failing a lot
 - Cross Python 2/3 compatibility with annotation arguments
 
-uncompyle6 2.11.5 2017-08-31
+2.11.5 2017-08-31
+=====================
 
 - Skeletal support for Python 3.7
 
-uncompyle6 2.11.4 2017-08-15
+2.11.4 2017-08-15
+=====================
 
 * scanner and parser now allow 3-part version string lookups,
   e.g. 2.7.1 We allow a float here, but if passed a string like '2.7'. or
@@ -237,7 +300,8 @@ uncompyle6 2.11.4 2017-08-15
 * Some PyPy tolerance in validate testing.
 * Some pyston tolerance
 
-uncompyle6 2.11.3 2017-08-09
+2.11.3 2017-08-09
+=====================
 
 Very minor changes
 
@@ -246,20 +310,24 @@ Very minor changes
 - use xdis opcode sets
 - xdis "exception match" is now "exception-match"
 
-uncompyle6 2.11.2 2017-07-09
+2.11.2 2017-07-09
+=====================
 
 - Start supporting Pypy 3.5 (5.7.1-beta)
 - use xdis 3.5.0's opcode sets and require xdis 3.5.0
 - Correct some Python 2.4-2.6 loop detection
 - guard against badly formatted bytecode
 
-uncompyle6 2.11.1 2017-06-25
+2.11.1 2017-06-25
+=====================
 
 - Python 3.x annotation and function signature fixes
 - Bump xdis version
 - Small pysource bug fixes
 
-uncompyle6 2.11.0 2017-06-18 Fleetwood
+2.11.0 2017-06-18 Fleetwood
+==================================
+
 - Major improvements in fragment tracking
   * Add nonterminal node in extractInfo
   * tag more offsets in expressions
@@ -269,14 +337,16 @@ uncompyle6 2.11.0 2017-06-18 Fleetwood
 - Fixes yet again for make_function node handling; document what's up here
 - Fix bug in snowflake Python 3.5 *args kwargs
 
-uncompyle6 2.10.1 2017-06-3 Marylin Frankel
+2.10.1 2017-06-3 Marylin Frankel
+========================================
 
 - fix some fragments parsing bugs
   - was returning the wrong type sometimes in deparse_code_around_offset()
   - capture function name in offsets
   - track changes to ifelstrmtr node from pysource into fragments
 
-uncompyle6 2.10.0 2017-05-30 Elaine Gordon
+2.10.0 2017-05-30 Elaine Gordon
+=======================================
 
 - Add fuzzy offset deparse look up
 - 3.6 bug fixes
@@ -296,19 +366,21 @@ uncompyle6 2.10.0 2017-05-30 Elaine Gordon
 - 2.3, 2.4 "if 1 .." fixes
 - 3.x annotation fixes
 
-uncompyle6 2.9.11 2017-04-06
+2.9.11 2017-04-06
+=====================
 
 - Better support for Python 3.5+ BUILD_MAP_UNPACK
 - Start 3.6 CALL_FUNCTION_EX support
 - Many decompilation bug fixes. (Many more remain). See ChangeLog
 
-uncompyle6 2.9.10 2017-02-25
+2.9.10 2017-02-25
+=====================
 
 - Python grammar rule fixes
 - Add ability to get grammar coverage on runs
 - Handle Python 3.6 opcode BUILD_CONST_KEYMAP
 
-uncompyle6 2.9.9 2016-12-16
+2.9.9 2016-12-16
 
 - Remaining Python 3.5 ops handled
   (this also means more Python 3.6 ops are handled)
@@ -318,7 +390,8 @@ uncompyle6 2.9.9 2016-12-16
 - Better control-flow detection
 - Code cleanups and misc bug fixes
 
-uncompyle6 2.9.8 2016-12-16
+2.9.8 2016-12-16
+====================
 
 - Better control-flow detection
 - pseudo instruction THEN in 2.x
@@ -331,7 +404,8 @@ uncompyle6 2.9.8 2016-12-16
 - verify call fixes for Python <= 2.4
 - more Python lint
 
-uncompyle6 2.9.7 2016-12-16
+2.9.7 2016-12-16
+====================
 
 - Start to handle 3.5/3.6 build_map_unpack_with_call
 - Some Python 3.6 bytecode to wordcode conversion fixes
@@ -341,7 +415,8 @@ uncompyle6 2.9.7 2016-12-16
 - some 3.2 compatibility
 - Better Python 3 control flow detection by adding Pseudo ELSE opcodes
 
-uncompyle6 2.9.6 2016-12-04
+2.9.6 2016-12-04
+====================
 
 - Shorten Python3 grammars with + and *
   this requires spark parser 1.5.1
@@ -349,7 +424,8 @@ uncompyle6 2.9.6 2016-12-04
   decompile accuracy. This too requires
   spark parser 1.5.1
 
-uncompyle6 2.9.6 2016-11-20
+2.9.6 2016-11-20
+====================
 
 - Correct MANIFEST.in
 - More AST grammar checking
@@ -366,7 +442,8 @@ uncompyle6 2.9.6 2016-11-20
 - Python 2 and 3 detect structure code is more similar
 - Handle Docstrings with embedded triple quotes (""")
 
-uncompyle6 2.9.5 2016-11-13
+2.9.5 2016-11-13
+====================
 
 - Fix Python 3 bugs:
   * improper while 1 else
@@ -376,13 +453,15 @@ uncompyle6 2.9.5 2016-11-13
 - Start grammar misparse checking
 
 
-uncompyle6 2.9.4 2016-11-02
+2.9.4 2016-11-02
+====================
 
 - Handle Python 3.x function annotations
 - track def keyword-parameter line-splitting in source code better
 - bump min xdis version to mask previous xdis bug
 
-uncompyle6 2.9.3 2016-10-26
+2.9.3 2016-10-26
+====================
 
 Release forced by incompatibility change in xdis 3.2.0.
 
@@ -397,7 +476,8 @@ Release forced by incompatibility change in xdis 3.2.0.
   * Handle 3.6 handle single and multiple fstring better
 
 
-uncompyle6 2.9.2 2016-10-15
+2.9.2 2016-10-15
+====================
 
 - use source-code line breaks to assist in where to break
   in tuples and maps
@@ -405,12 +485,14 @@ uncompyle6 2.9.2 2016-10-15
 - Fix some Python 2.6 and below bugs
 - DRY fragments.py code a little
 
-uncompyle6 2.9.1 2016-10-09
+2.9.1 2016-10-09
+====================
 
 - Improved Python 1.5 decompiling
 - Handle old-style pre Python 2.2 classes
 
-uncompyle6 2.9.0 2016-10-09
+2.9.0 2016-10-09
+====================
 
 - Use xdis 3.0.0 protocol load_module.
   this Forces change in requirements.txt and _pkg_info_.py
@@ -420,7 +502,8 @@ uncompyle6 2.9.0 2016-10-09
 - Fix bug with -t ...  Wasn't showing source text when -t option was given
 - Fix 2.1-2.6 bug in list comprehension
 
-uncompyle6 2.8.4 2016-10-08
+2.8.4 2016-10-08
+====================
 
 - Python 3 disassembly bug fixes
 - Python 3.6 fstring bug fixes (from moagstar)
@@ -428,7 +511,8 @@ uncompyle6 2.8.4 2016-10-08
 - COME_FROM suffixes added in Python3
 - use .py extension in verification disassembly
 
-uncompyle6 2.8.3 2016-09-11 live from NYC!
+2.8.3 2016-09-11 live from NYC!
+=======================================
 
 NOTE: this is possibly the last release before a major reworking of
 control-flow structure detection is done.
@@ -456,14 +540,16 @@ control-flow structure detection is done.
 - bump xdis requirement so we can deparse dropbox 2.5 code
 - Added H. Goebel's changes before 2.4 in DECOMPYLE-2.4-CHANGELOG.txt
 
-uncompyle6 2.8.2 2016-08-29
+2.8.2 2016-08-29
+====================
 
 - Handle Python 3.6 format string conversions !r, !s, !a
 - Start to handle 3.1 bytecode
 - Fix some PyPy translation bugs
 - We now only handle 3.6.0a3+ since that is incompatible with 3.6 before that
 
-uncompyle6 2.8.1 2016-08-20
+2.8.1 2016-08-20
+====================
 
 - Add Python 2.2 decompilation
 
@@ -471,7 +557,8 @@ uncompyle6 2.8.1 2016-08-20
  * PyPy LOOKUP_METHOD bug
  * Python 3.6 FORMAT_VALUE handles expressions now
 
-uncompyle6 2.8.0 2016-08-03
+2.8.0 2016-08-03
+====================
 
 - Start Python 3.6 support (moagstar)
   more work on PEP 498 needed
@@ -482,20 +569,23 @@ uncompyle6 2.8.0 2016-08-03
 - better grammar and semantic action segregation based
   on python bytecode version
 
-uncompyle6 2.7.1 2016-07-26
+2.7.1 2016-07-26
+====================
 
 - PyPy bytecodes for 2.7 and 3.2 added
 - Instruction formatting improved slightly
 - 2.7 bytecode "continue" bug fixed
 
-uncompyle6 2.7.0 2016-07-15
+2.7.0 2016-07-15
+====================
 
 - Many Syntax and verification bugs removed
   tested on standard libraries from 2.3.7 to 3.5.1
   and they all decompile and verify fine.
   I'm sure there are more bugs though.
 
-uncompyle6 2.6.2 2016-07-11 Manhattenhenge
+2.6.2 2016-07-11 Manhattenhenge
+=======================================
 
 - Extend bytecodes back to 2.3
 - Fix bugs:
@@ -504,13 +594,15 @@ uncompyle6 2.6.2 2016-07-11 Manhattenhenge
   * continue statements
 - DRY and segregate grammar more
 
-uncompyle6 2.6.1 2016-07-08
+2.6.1 2016-07-08
+====================
 
 - Go over Python 2.5 bytecode deparsing
   all library programs now deparse
 - Fix a couple bugs in 2.6 deparsing
 
-uncompyle6 2.6.0 2016-07-07
+2.6.0 2016-07-07
+====================
 
 - Improve Python 2.6 bytecode deparsing:
   stdlib now will deparse something
@@ -519,7 +611,8 @@ uncompyle6 2.6.0 2016-07-07
 - Fix bug in installing uncompyle6 script
 - Doc improvements
 
-uncompyle6 2.5.0 2016-06-22  Summer Solstice
+2.5.0 2016-06-22 Summer Solstice
+========================================
 
 - Much better Python 3.2-3.5 coverage.
   3.4.6 is probably the best;3.2 and 3.5 are weaker
@@ -528,7 +621,8 @@ uncompyle6 2.5.0 2016-06-22  Summer Solstice
 - Better fragment offset tracking
 - Some (much-needed) code refactoring
 
-uncompyle6 2.4.0 2016-05-18 (in memory of Lewis Bernstein)
+2.4.0 2016-05-18 (in memory of Lewis Bernstein)
+===========================================================
 
 - Many Python 3 bugs fixed:
   * Python 3.2 to 3.5 libraries largely
@@ -543,7 +637,8 @@ uncompyle6 2.4.0 2016-05-18 (in memory of Lewis Bernstein)
   * handle complex number unmarshaling
   * Running on Python 2 to works on Python 3.5 bytecodes now
 
-uncompyle6 2.3.5 and 2.3.6 2016-05-14
+2.3.5 and 2.3.6 2016-05-14
+=================================
 
 - Python 2 class decorator fix (thanks to Tey)
 - Fix fragment parsing bugs
@@ -555,20 +650,23 @@ uncompyle6 2.3.5 and 2.3.6 2016-05-14
 - Correct history based on info from Dan Pascu
 - Fix up pip packaging, ugh.
 
-uncompyle6 2.3.4 2016-05-5
+2.3.4 2016-05-5
+===================
 
 - More Python 3.5 parsing bugs addressed
 - decompiling Python 3.5 from other Python versions works
 - test from Python 3.2
 - remove "__module__ = __name__" in 3.0 <= Python 3.2
 
-uncompyle6 2.3.3 2016-05-3
+2.3.3 2016-05-3
+===================
 
 - Fix bug in running uncompyle6 script on Python 3
 - Speed up performance on deparsing long lists by grouping in chunks of 32 and 256 items
 - DRY Python expressions between Python 2 and 3
 
-uncompyle6 2.3.2 2016-05-1
+2.3.2 2016-05-1
+===================
 
 - Add --version option standalone scripts
 - Correct License information in package
@@ -577,17 +675,20 @@ uncompyle6 2.3.2 2016-05-1
   specific grammar code
 - Fix bug in 3.5+ constant map parsing
 
-uncompyle6 2.3.0, 2.3.1 2016-04-30
+2.3.0, 2.3.1 2016-04-30
+=============================
 
 - Require spark_parser  >= 1.1.0
 
-uncompyle6 2.2.0 2016-04-30
+2.2.0 2016-04-30
+====================
 
 - Spark is no longer here but pulled separate package spark_parse
 - Python 3 parsing fixes
 - More tests
 
-uncompyle6 2.2.0 2016-04-02
+2.2.0 2016-04-02
+====================
 
 - Support single-mode (in addition to exec-mode) compilation
 - Start to DRY Python 2 and Python 3 grammars
@@ -595,7 +696,8 @@ uncompyle6 2.2.0 2016-04-02
 - Fix bug in uncomplye6 -d and -r options (via lelicopter)
 
 
-uncompyle6 2.1.3 2016-01-02
+2.1.3 2016-01-02
+====================
 
 - Limited support for decompiling Python 3.5
 - Improve Python 3 class deparsing
@@ -604,18 +706,21 @@ uncompyle6 2.1.3 2016-01-02
 - increase test coverage
 - fix misc small bugs and some improvements
 
-uncompyle6 2.1.2 2015-12-31
+2.1.2 2015-12-31
+====================
 
 - Fix cross-version Marshal loading
 - Handle Python 3.3 . dotted class names
 - Limited 3.5 support: allows deparsing other versions
 - Refactor code more, misc bug fixes
 
-uncompyle6 2.1.1 2015-12-27
+2.1.1 2015-12-27
+====================
 
 - packaging issues
 
-uncompyle6 2.1.0 2015-12-27
+2.1.0 2015-12-27
+====================
 
 - Python 3.x deparsing much more solid
 - Better cross-version deparsing
@@ -624,7 +729,8 @@ Some bugs squashed while other run rampant. Some code cleanup while
 much more is yet needed. More tests added, but many more are needed.
 
 
-uncompyle6 2.0.0 2015-12-11
+2.0.0 2015-12-11
+====================
 
 Changes from uncompyle2
 

__pkginfo__.py

@@ -23,7 +23,7 @@
 
 # Things that change more often go here.
 copyright   = """
-Copyright (C) 2015-2018 Rocky Bernstein <rb@dustyfeet.com>.
+Copyright (C) 2015-2019 Rocky Bernstein <rb@dustyfeet.com>.
 """
 
 classifiers =  ['Development Status :: 5 - Production/Stable',
@@ -57,7 +57,7 @@ entry_points       = {
     ]}
 ftp_url            = None
 install_requires   = ['spark-parser >= 1.8.7, < 1.9.0',
-                      'xdis >= 3.8.8, < 3.9.0']
+                      'xdis >= 4.0.0, < 4.1.0']
 
 license            = 'GPL3'
 mailing_list       = 'python-debugger@googlegroups.com'

admin-tools/how-to-make-a-release.md

@@ -30,9 +30,9 @@
 
     $ make ChangeLog
 
-#  Update NEWS from ChangeLog:
+#  Update NEWS.md from ChangeLog:
 
-    $ emacs NEWS
+    $ emacs NEWS.md
     $ make check
     $ git commit --amend .
     $ git push   # get CI testing going early
@@ -58,7 +58,8 @@
     $ git tag release-python-2.4-$VERSION
 
     $ . ./admin-tools/make-dist-newer.sh
-    $ git tag release-$VERSION
+
+Goto https://github.com/rocky/python-uncompyle6/releases
 
 # Upload single package and look at Rst Formating
 

admin-tools/pyenv-newer-versions

@@ -5,4 +5,4 @@ if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
     echo "This script should be *sourced* rather than run directly through bash"
     exit 1
 fi
-export PYVERSIONS='3.5.5 3.6.6 3.7.1 2.6.9 3.3.7 2.7.14 3.2.6 3.1.5 3.4.8'
+export PYVERSIONS='3.2.6 3.6.8 3.7.2 2.6.9 3.3.7 2.7.15 3.2.6 3.1.5 3.4.8'

pytest/test_docstring.py

@@ -47,7 +47,7 @@ class PrintFake:
             out = out[:-self.pending_newlines]
         self.f.write(out)
     def println(self, *data):
-        if data and not(len(data) == 1 and data[0] ==''):
+        if data and not(len(data) == 1 and data[0] == ''):
             self.write(*data)
         self.pending_newlines = max(self.pending_newlines, 1)
         return

pytest/test_grammar.py

@@ -7,7 +7,7 @@ def test_grammar():
 
     def check_tokens(tokens, opcode_set):
         remain_tokens = set(tokens) - opcode_set
-        remain_tokens = set([re.sub('_\d+$','', t) for t in remain_tokens])
+        remain_tokens = set([re.sub(r'_\d+$','', t) for t in remain_tokens])
         remain_tokens = set([re.sub('_CONT$','', t) for t in remain_tokens])
         remain_tokens = set(remain_tokens) - opcode_set
         assert remain_tokens == set([]), \
@@ -18,15 +18,19 @@ def test_grammar():
      right_recursive, dup_rhs) = p.check_sets()
 
     # We have custom rules that create the below
-    expect_lhs = set(['pos_arg', 'get_iter', 'attribute'])
+    expect_lhs = set(['pos_arg', 'attribute'])
+    if PYTHON_VERSION < 3.8:
+        expect_lhs.add('get_iter')
+
 
     unused_rhs = set(['list', 'mkfunc',
                       'mklambda',
                       'unpack',])
+
     expect_right_recursive = set([('designList',
                                    ('store', 'DUP_TOP', 'designList'))])
 
-    if PYTHON_VERSION != 3.7:
+    if PYTHON_VERSION < 3.7:
         unused_rhs.add('call')
 
     if PYTHON_VERSION > 2.6:
@@ -46,6 +50,7 @@ def test_grammar():
             unused_rhs.add("mkfunc_annotate")
             unused_rhs.add("dict_comp")
             unused_rhs.add("classdefdeco1")
+            unused_rhs.add("tryelsestmtl")
             if PYTHON_VERSION >= 3.5:
                 expect_right_recursive.add((('l_stmts',
                                              ('lastl_stmt', 'come_froms', 'l_stmts'))))
@@ -60,7 +65,11 @@ def test_grammar():
         expect_lhs.add('kwarg')
 
     assert expect_lhs == set(lhs)
-    assert unused_rhs == set(rhs)
+
+    # FIXME
+    if PYTHON_VERSION != 3.8:
+        assert unused_rhs == set(rhs)
+
     assert expect_right_recursive == right_recursive
 
     expect_dup_rhs = frozenset([('COME_FROM',), ('CONTINUE',), ('JUMP_ABSOLUTE',),

pytest/validate.py

@@ -79,7 +79,7 @@ def are_instructions_equal(i1, i2):
 
     :return: True if the two instructions are approximately equal, otherwise False.
     """
-    result = (1==1
+    result = (1 == 1
         and i1.opname == i2.opname
         and i1.opcode == i2.opcode
         and i1.arg == i2.arg

setup.py

@@ -6,7 +6,7 @@ import sys
 SYS_VERSION = sys.version_info[0:2]
 if not ((2, 4) <= SYS_VERSION  <= (2, 7)):
     mess = "Python Release 2.4 .. 2.7 are supported in this code branch."
-    if ((3, 2) <= SYS_VERSION  <= (3, 7)):
+    if ((3, 2) <= SYS_VERSION  <= (3, 8)):
         mess += ("\nFor your Python, version %s, use the master code/branch." %
                  sys.version[0:3])
     else:

test/Makefile

@@ -33,42 +33,49 @@ check-2.4 check-2.5 check-2.6 check-2.7: check-bytecode-2 check-bytecode-3 check
 
 #: Run working tests from Python 3.0
 check-3.0: check-bytecode
-	$(PYTHON) test_pythonlib.py --bytecode-3.0 --weak-verify $(COMPILE)
 	$(PYTHON) test_pythonlib.py --bytecode-3.0-run --verify-run
+	$(PYTHON) test_pythonlib.py --bytecode-3.0 --weak-verify $(COMPILE)
 
 #: Run working tests from Python 3.1
 check-3.1: check-bytecode
-	$(PYTHON) test_pythonlib.py --bytecode-3.1 --weak-verify $(COMPILE)
 	$(PYTHON) test_pythonlib.py --bytecode-3.1-run --verify-run
+	$(PYTHON) test_pythonlib.py --bytecode-3.1 --weak-verify $(COMPILE)
 
 #: Run working tests from Python 3.2
 check-3.2: check-bytecode
-	$(PYTHON) test_pythonlib.py --bytecode-3.2 --weak-verify $(COMPILE)
 	$(PYTHON) test_pythonlib.py --bytecode-3.2-run --verify-run
+	$(PYTHON) test_pythonlib.py --bytecode-3.2 --weak-verify $(COMPILE)
 
 #: Run working tests from Python 3.3
 check-3.3: check-bytecode
-	$(PYTHON) test_pythonlib.py --bytecode-3.3 --weak-verify $(COMPILE)
 	$(PYTHON) test_pythonlib.py --bytecode-3.3-run --verify-run
+	$(PYTHON) test_pythonlib.py --bytecode-3.3 --weak-verify $(COMPILE)
 
 #: Run working tests from Python 3.4
 check-3.4: check-bytecode check-3.4-ok check-2.7-ok
-	$(PYTHON) test_pythonlib.py --bytecode-3.4 --weak-verify $(COMPILE)
 	$(PYTHON) test_pythonlib.py --bytecode-3.4-run --verify-run
+	$(PYTHON) test_pythonlib.py --bytecode-3.4 --weak-verify $(COMPILE)
 
 #: Run working tests from Python 3.5
 check-3.5: check-bytecode
-	$(PYTHON) test_pythonlib.py --bytecode-3.5 --weak-verify $(COMPILE)
 	$(PYTHON) test_pythonlib.py --bytecode-3.5-run --verify-run
+	$(PYTHON) test_pythonlib.py --bytecode-3.5 --weak-verify $(COMPILE)
 
 #: Run working tests from Python 3.6
 check-3.6: check-bytecode
+	$(PYTHON) test_pythonlib.py --bytecode-3.6-run --verify-run
 	$(PYTHON) test_pythonlib.py --bytecode-3.6 --weak-verify $(COMPILE)
 
 #: Run working tests from Python 3.7
 check-3.7: check-bytecode
+	$(PYTHON) test_pythonlib.py --bytecode-3.7-run --verify-run
 	$(PYTHON) test_pythonlib.py --bytecode-3.7 --weak-verify $(COMPILE)
 
+#: Run working tests from Python 3.8
+check-3.8: check-bytecode
+	$(PYTHON) test_pythonlib.py --bytecode-3.8-run --verify-run
+	$(PYTHON) test_pythonlib.py --bytecode-3.8 --weak-verify $(COMPILE)
+
 # FIXME
 #: this is called when running under pypy3.5-5.8.0 or pypy2-5.6.0
 5.8 5.6:
@@ -89,8 +96,10 @@ check-bytecode-2:
 #: Check deparsing bytecode 3.x only
 check-bytecode-3:
 	$(PYTHON) test_pythonlib.py --bytecode-3.0 \
-       --bytecode-3.1 --bytecode-3.2 --bytecode-3.3 \
-       --bytecode-3.4 --bytecode-3.5 --bytecode-3.6 --bytecode-pypy3.2
+	--bytecode-3.1 --bytecode-3.2 --bytecode-3.3 \
+	--bytecode-3.4 --bytecode-3.5 --bytecode-3.6 \
+        --bytecode-3.7 --bytecode-3.8 \
+	--bytecode-pypy3.2
 
 #: Check deparsing on selected bytecode 3.x
 check-bytecode-3-short:
@@ -139,6 +148,7 @@ check-bytecode-2.3:
 
 #: Check deparsing Python 2.4
 check-bytecode-2.4:
+	$(PYTHON) test_pythonlib.py --bytecode-2.4-run --verify-run
 	$(PYTHON) test_pythonlib.py --bytecode-2.4
 
 #: Check deparsing Python 2.5
@@ -214,43 +224,43 @@ grammar-coverage-3.6:
 
 #: Check deparsing Python 2.6
 check-bytecode-2.6:
-	$(PYTHON) test_pythonlib.py --bytecode-2.6 --weak-verify
 	$(PYTHON) test_pythonlib.py --bytecode-2.6-run --verify-run
+	$(PYTHON) test_pythonlib.py --bytecode-2.6 --weak-verify
 
 #: Check deparsing Python 2.7
 check-bytecode-2.7:
-	$(PYTHON) test_pythonlib.py --bytecode-2.7 --weak-verify
 	$(PYTHON) test_pythonlib.py --bytecode-2.7-run --verify-run
+	$(PYTHON) test_pythonlib.py --bytecode-2.7 --weak-verify
 
 #: Check deparsing Python 3.0
 check-bytecode-3.0:
-	$(PYTHON) test_pythonlib.py --bytecode-3.0 --weak-verify
 	$(PYTHON) test_pythonlib.py --bytecode-3.0-run --verify-run
+	$(PYTHON) test_pythonlib.py --bytecode-3.0 --weak-verify
 
 #: Check deparsing Python 3.1
 check-bytecode-3.1:
-	$(PYTHON) test_pythonlib.py --bytecode-3.1 --weak-verify
 	$(PYTHON) test_pythonlib.py --bytecode-3.1-run --verify-run
+	$(PYTHON) test_pythonlib.py --bytecode-3.1 --weak-verify
 
 #: Check deparsing Python 3.2
 check-bytecode-3.2:
-	$(PYTHON) test_pythonlib.py --bytecode-3.2 --weak-verify
 	$(PYTHON) test_pythonlib.py --bytecode-3.2-run --verify-run
+	$(PYTHON) test_pythonlib.py --bytecode-3.2 --weak-verify
 
 #: Check deparsing Python 3.3
 check-bytecode-3.3:
-	$(PYTHON) test_pythonlib.py --bytecode-3.3 --weak-verify
 	$(PYTHON) test_pythonlib.py --bytecode-3.3-run --verify-run
+	$(PYTHON) test_pythonlib.py --bytecode-3.3 --weak-verify
 
 #: Check deparsing Python 3.4
 check-bytecode-3.4:
-	$(PYTHON) test_pythonlib.py --bytecode-3.4 --weak-verify
 	$(PYTHON) test_pythonlib.py --bytecode-3.4-run --verify-run
+	$(PYTHON) test_pythonlib.py --bytecode-3.4 --weak-verify
 
 #: Check deparsing Python 3.5
 check-bytecode-3.5:
-	$(PYTHON) test_pythonlib.py --bytecode-3.5 --weak-verify
 	$(PYTHON) test_pythonlib.py --bytecode-3.5-run --verify-run
+	$(PYTHON) test_pythonlib.py --bytecode-3.5 --weak-verify
 
 #: Check deparsing Python 3.6
 check-bytecode-3.6:
@@ -261,6 +271,10 @@ check-bytecode-3.6:
 check-bytecode-3.7:
 	$(PYTHON) test_pythonlib.py --bytecode-3.7 --weak-verify
 
+#: Check deparsing Python 3.8
+check-bytecode-3.8:
+	$(PYTHON) test_pythonlib.py --bytecode-3.8 --weak-verify
+
 #: short tests for bytecodes only for this version of Python
 check-native-short:
 	$(PYTHON) test_pythonlib.py --bytecode-$(PYTHON_VERSION) --weak-verify $(COMPILE)

test/add-test.py

@@ -4,12 +4,19 @@
 import os, sys, py_compile
 assert len(sys.argv) >= 2
 version = sys.version[0:3]
-for path in sys.argv[1:]:
+if sys.argv[1] == '--run':
+    suffix = '_run'
+    py_source = sys.argv[2:]
+else:
+    suffix = ''
+    py_source = sys.argv[1:]
+
+for path in py_source:
     short = os.path.basename(path)
     if hasattr(sys, 'pypy_version_info'):
-        cfile =  "bytecode_pypy%s/%s" % (version, short) + 'c'
+        cfile =  "bytecode_pypy%s%s/%s" % (version, suffix, short) + 'c'
     else:
-        cfile =  "bytecode_%s/%s" % (version, short) + 'c'
+        cfile =  "bytecode_%s%s/%s" % (version, suffix, short) + 'c'
     print("byte-compiling %s to %s" % (path, cfile))
     py_compile.compile(path, cfile)
     if isinstance(version, str) or version >= (2, 6, 0):

test/decompyle/test_augmentedAssign.py

@@ -18,7 +18,7 @@ a *= b; print a  # a = a*b = 2
 a -= a; print a  # a = a-a = 0
 a += 7*3; print a  # == 21
 
-l= [1,2,3]
+l = [1,2,3]
 l[1] *= 3;    print l[1]; # 6
 l[1][2][3] = 7
 l[1][2][3] *= 3;

test/dis-compare.py

@@ -75,7 +75,7 @@ def get_srcdir():
 src_dir = get_srcdir()
 os.chdir(src_dir)
 
-files=[
+files = [
     'if',
     'ifelse',
     # 'keyword',

test/simple_source/branching/02_ifelse_lambda.py

@@ -1,7 +1,7 @@
 # We have to do contortions here because
 # lambda's have to be more or less on a line
 
-f = lambda x: 1 if x<2 else 3
+f = lambda x: 1 if x < 2 else 3
 assert f(3) == 3
 assert f(1) == 1
 

test/simple_source/bug24/03_iftrue.py

@@ -0,0 +1,10 @@
+# Python 2.4 (and before?) bug in handling unconditional "else if true"
+# Doesn't occur in Python > 2.4
+# From Issue #187
+def unconditional_if_true_24(foo):
+    if not foo:
+        pass
+    elif 1:
+        pass
+    else:
+        return None

test/simple_source/bug27+/05_while_elif.py

@@ -0,0 +1,16 @@
+# Bug in Python 2.7. Bug is bytecode for while loop having
+# two consecutive JUMP_BACKS at the end of 'elif' and 'while'
+# to the same place
+def PreprocessConditionalStatement(self, IfList, ReplacedLine):
+    while self:
+        if self.__Token:
+            x = 1
+        elif not IfList:
+            if self <= 2:
+                continue
+            RegionSizeGuid = 3
+            if not RegionSizeGuid:
+                RegionLayoutLine = 5
+                continue
+            RegionLayoutLine = self.CurrentLineNumber
+    return 1

test/simple_source/bug27+/06_raise.py

@@ -0,0 +1,22 @@
+# Bug in Python 2.7 is code creating a (useless) JUMP_ABSOLUTE to the instruction right after
+# the "raise" which causes the
+
+# RUNNABLE!
+def testit(a, b):
+    if a:
+        if not b:
+            raise AssertionError("test JUMP_ABSOLUTE to next instruction")
+
+def testit2(a, b):
+    if a:
+        if not b:
+            raise AssertionError("test with dead code after raise")
+            x = 10
+
+testit(False, True)
+testit(False, False)
+testit(True, True)
+
+testit2(False, True)
+testit2(False, False)
+testit2(True, True)

test/simple_source/bug275/01_while1.py

@@ -3,7 +3,7 @@
 #  while1stmt        ::= SETUP_LOOP l_stmts_opt JUMP_BACK COME_FROM
 #  while1stmt        ::= SETUP_LOOP l_stmts_opt CONTINUE COME_FROM
 # which is included in later code generation
-ms=0
-if ms==1:
+ms = 0
+if ms == 1:
     while 1:
         pass

test/simple_source/bug31/01_try_else.py

@@ -0,0 +1,24 @@
+# Bug based on 2.7 test_itertools.py but mis-decompiled in Python 3.x bytecode
+# The bug is confusing "try/else" with "try" as a result of the loop which causes
+# the end of the except to jump back to the beginning of the loop, outside of the
+# try. In 3.x we not distinguising this jump out of the loop with a jump to the
+# end of the "try".
+
+# RUNNABLE!
+def testit(stmts):
+
+    # Bug was confusing When the except jumps back to the beginning of the block
+    # to the beginning of this for loop
+    x = 1
+    results = []
+    for stmt in stmts:
+        try:
+            x = eval(stmt)
+        except SyntaxError:
+            results.append(1)
+        else:
+            results.append(x)
+    return results
+
+results = testit(["1 + 2", "1 +"])
+assert results == [3, 1], "try with else failed"

test/simple_source/bug36/00_if_elif.py

@@ -0,0 +1,16 @@
+# Self-checking test.
+# Python 3 bug in not detecting the end bounds of if elif.
+def testit(b):
+    if b == 1:
+        a = 1
+    elif b == 2:
+        a = 2
+    else:
+        a = 4
+
+    return a
+
+for x in (1, 2, 4):
+    x = testit(x)
+    assert x is not None, "Should have returned a value, not None"
+    assert x == x

test/simple_source/bug36/00_return_return_bug.py

@@ -0,0 +1,10 @@
+# Bug in 3.6 and above.
+#Not detecting 2nd return is outside of
+# if/then. Fix was to ensure COME_FROM
+def return_return_bug(foo):
+    if foo == 'say_hello':
+        return "hello"
+    return "world"
+
+assert return_return_bug('say_hello') == 'hello'
+assert return_return_bug('world') == 'world'

test/simple_source/bug36/01_fstring.py

@@ -1,4 +1,5 @@
-# Self-checking 3.6+ string interpolation tests
+# Self-checking test.
+# String interpolation tests
 
 var1 = 'x'
 var2 = 'y'

test/simple_source/bug36/01_if_and_if_bug.py

@@ -0,0 +1,16 @@
+# Bug in 3.6 was not taking "else" branch after compond "if"
+# In earlier versions we had else detection needed here.
+
+def f(a, b, c):
+    if a and b:
+        x = 1
+    else:
+        x = 2
+    if c:
+        x = 3
+    return(x)
+
+assert f(True, True, True) == 3
+assert f(True, True, False) == 1
+assert f(True, False, True) == 3
+assert f(True, False, False) == 2

test/simple_source/bug36/03_if_try.py

@@ -0,0 +1,8 @@
+# The bug in python 3.6+ was in parsing that we
+# add END_IF_THEN and using that inside "return results"
+def whcms_license_info(md5hash, datahash, results):
+    if md5hash == datahash:
+        try:
+            return md5hash
+        except:
+            return results

test/simple_source/bug36/07_kwargs.py

@@ -0,0 +1,9 @@
+# Bug in Python 3.6 and 3.7 was getting comma before **kw
+
+def fn(arg, *, kwarg='test', **kw):
+    assert arg == 1
+    assert kwarg == 'testing'
+    assert kw['foo'] == 'bar'
+
+
+fn(1, kwarg='testing', foo='bar')

test/simple_source/bug37/01_chained_compare.py

@@ -0,0 +1,19 @@
+# From Python 3.7 pickle.py
+# Bug was different code generation for chained comparisons than prior Python versions
+
+
+def chained_compare_a(protocol):
+    if not 0 <= protocol <= 7:
+        raise ValueError("pickle protocol must be <= %d" % 7)
+
+def chained_compare_b(a, obj):
+    if a:
+        if -0x80000000 <= obj <= 0x7fffffff:
+            return 5
+
+chained_compare_a(3)
+try:
+    chained_compare_a(8)
+except ValueError:
+    pass
+chained_compare_b(True, 0x0)

test/simple_source/looping/10_for.py

@@ -1,8 +1,17 @@
+# Self-checking test.
 # Tests:
-#  forstmt ::= SETUP_LOOP expr _for store
-#              for_block POP_BLOCK COME_FROM
-for a in [1]:
-    c = 2
+#  for ::= SETUP_LOOP expr for_iter store
+#          for_block POP_BLOCK COME_FROM
+# In 3.8+
+#  for ::= expr for_iter store
+#          for_block POP_BLOCK COME_FROM
 
-for a in range(2):
-    c = 2
+c = 0
+for a in [1]:
+    c += a
+assert c == 1, c
+
+for a in range(3):
+    c += a
+
+assert c == 4, c

test/simple_source/operation_logic/10_mixed_boolean.py

@@ -1,12 +1,26 @@
+# Self-checking test.
+# Mixed boolean expresions
+
 b = True
+assert b, 'b = True'
 c = False
+assert not c, 'c = False'
 d = True
 a = b and c or d
+assert a, 'b and c or d'
 a = (b or c) and d
+assert a, '(b or c) and d'
 a = b or c or d
+assert  a, 'b or c or d'
 a = b and c and d
+assert not a, 'b and c and d'
 a = b or c and d
+assert a
 a = b and (c or d)
+assert a
 a = b and c or d
+assert a
 a = (b or c and d) and b
+assert a
 a = (b or c and d or a) and b
+assert a

test/simple_source/slice/02_slice.py

@@ -1,8 +1,15 @@
+# Self-checking test.
 # Tests of Python slice operators
 
 ary = [1,2,3,4,5]
+
 # Forces BUILD_SLICE 2 on 3.x
-ary[:2]
-ary[2:]
+a = ary[:2]
+assert a == [1, 2]
+
+a = ary[2:]
+assert a == [3, 4, 5]
+
 # Forces BUILD_SLICE 3
-ary[1:4:2]
+a = ary[1:4:2]
+assert a == [2, 4]

test/simple_source/stmts/01_augmented_assign.py

@@ -20,7 +20,7 @@ a *= b; # print a  # a = a*b = 2
 a -= a; # print a  # a = a-a = 0
 a += 7*3; # print a  # == 21
 
-l= [1,2,3]
+l = [1,2,3]
 l[1] *= 3;  #  print l[1]; # 6
 l[1][2][3] = 7
 l[1][2][3] *= 3;

test/simple_source/stmts/10_del.py

@@ -28,7 +28,7 @@ def foo():
 z = {}
 
 def a():
-    b =1
+    b = 1
     global z
     del z
     def b(y):

test/stdlib/runtests.sh

@@ -115,13 +115,20 @@ case $PYVERSION in
 	                          # See test/simple_source/bug27+/05_not_unconditional.py
 	    [test_memoryio.py]=1 # FIX
 	    [test_multiprocessing.py]=1 # On uncompyle2, taks 24 secs
-	    [test_pep352.py]=1 # ?
+	    [test_pep352.py]=1  # ?
+	    [test_posix.py]=1   # Bug in try-else detection inside test_initgroups()
+	                        # Deal with when we have better flow-control detection
+	    [test_pwd.py]=1     # Takes too long
+	    [test_pty.py]=1
+	    [test_queue.py]=1   # Control flow?
+	    [test_re.py]=1      # Probably Control flow?
 	    [test_select.py]=1  # Runs okay but takes 11 seconds
 	    [test_socket.py]=1  # Runs ok but takes 22 seconds
 	    [test_subprocess.py]=1 # Runs ok but takes 22 seconds
 	    [test_sys_settrace.py]=1 # Line numbers are expected to be different
 	    [test_strtod.py]=1 # FIX
 	    [test_traceback.py]=1 # Line numbers change - duh.
+	    [test_types.py]=1     # try/else confusions
 	    [test_unicode.py]=1  # Too long to run 11 seconds
 	    [test_xpickle.py]=1 # Runs ok but takes 72 seconds
 	    [test_zipfile64.py]=1  # Runs ok but takes 204 seconds

test/test_pyenvlib.py

@@ -35,7 +35,7 @@ python_versions = [v for v in magics.python_versions if
 # FIXME: we should remove Python versions that we don't support.
 # These include Jython, and Python bytecode changes pre release.
 
-TEST_VERSIONS=(
+TEST_VERSIONS = (
                'pypy-2.4.0', 'pypy-2.6.1',
                'pypy-5.0.1', 'pypy-5.3.1', 'pypy3.5-5.7.1-beta',
                'native') + tuple(python_versions)

test/test_pythonlib.py

@@ -79,7 +79,7 @@ for  vers in (2.7, 3.4, 3.5, 3.6):
 for  vers in (1.3, 1.4, 1.5,
               2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7,
               3.0, 3.1, 3.2, 3.3,
-              3.4, 3.5, 3.6, 3.7, 'pypy3.2', 'pypy2.7'):
+              3.4, 3.5, 3.6, 3.7, 3.8, 'pypy3.2', 'pypy2.7'):
     bytecode = "bytecode_%s" % vers
     key = "bytecode-%s" % vers
     test_options[key] =  (bytecode, PYC, bytecode, vers)

tox.ini

@@ -3,7 +3,7 @@
 [flake8]
 exclude = .tox,./build,./trepan/processor/command/tmp
 filename = *.py
-ignore = C901,E113,E121,E122,E123,E124,E125,E126,E127,E128,E129,E201,E202,E203,E221,E222,E225,E226,E241,E242,E251,E261,E271,E272,E302,E401,E501,F401,E701,E702
+ignore = C901,E113,E121,E122,E123,E124,E125,E126,E127,E128,E129,E201,E202,E203,E221,E222,E225,E226,E241,E242,E251,E261,E271,E272,E302,E401,E402,E501,F401,E701,E702
 
 [tox]
 envlist = py27, py34, pypy

uncompyle6/bin/uncompile.py

@@ -1,7 +1,7 @@
 #!/usr/bin/env python
 # Mode: -*- python -*-
 #
-# Copyright (c) 2015-2017 by Rocky Bernstein
+# Copyright (c) 2015-2017, 2019 by Rocky Bernstein
 # Copyright (c) 2000-2002 by hartmut Goebel <h.goebel@crazy-compilers.com>
 #
 import sys, os, getopt, time
@@ -29,7 +29,8 @@ Options:
                     -> /tmp/bla/fasel.pyc_dis, /tmp/bar/foo.pyc_dis
                   uncompyle6 -o /tmp /usr/lib/python1.5
                     -> /tmp/smtplib.pyc_dis ... /tmp/lib-tk/FixTk.pyc_dis
-  -c <file>     attempts a disassembly after compiling <file>
+  --compile | -c <python-file>
+                attempts a decompilation after compiling <python-file>
   -d            print timestamps
   -p <integer>  use <integer> number of processes
   -r            recurse directories looking for .pyc and .pyo files
@@ -42,10 +43,10 @@ Options:
   --help        show this message
 
 Debugging Options:
-  --asm     -a  include byte-code         (disables --verify)
-  --grammar -g  show matching grammar
-  --tree    -t  include syntax tree       (disables --verify)
-  --tree++      add template rules to --tree when possible
+  --asm     | -a  include byte-code       (disables --verify)
+  --grammar | -g  show matching grammar
+  --tree    | -t  include syntax tree     (disables --verify)
+  --tree++        add template rules to --tree when possible
 
 Extensions of generated files:
   '.pyc_dis' '.pyo_dis'   successfully decompiled (and verified if --verify)
@@ -60,10 +61,7 @@ from uncompyle6.main import main, status_msg
 from uncompyle6.version import VERSION
 
 def usage():
-    print("""usage:
-    %s [--verify | --weak-verify ] [--asm] [--tree[+]] [--grammar] [-o <path>] FILE|DIR...
-   %s [--help | -h | --version | -V]
-"""  % (program, program))
+    print(__doc__)
     sys.exit(1)
 
 
@@ -77,13 +75,13 @@ def main_bin():
     numproc = 0
     outfile = '-'
     out_base = None
-    codes = []
+    source_paths = []
     timestamp = False
     timestampfmt = "# %Y.%m.%d %H:%M:%S %Z"
 
     try:
-        opts, files = getopt.getopt(sys.argv[1:], 'hagtdrVo:c:p:',
-                                    'help asm grammar linemaps recurse '
+        opts, pyc_paths = getopt.getopt(sys.argv[1:], 'hac:gtdrVo:p:',
+                                    'help asm compile= grammar linemaps recurse '
                                     'timestamp tree tree+ '
                                     'fragments verify verify-run version '
                                     'weak-verify '
@@ -125,8 +123,8 @@ def main_bin():
             outfile = val
         elif opt in ('--timestamp', '-d'):
             timestamp = True
-        elif opt == '-c':
-            codes.append(val)
+        elif opt in ('--compile', '-c'):
+            source_paths.append(val)
         elif opt == '-p':
             numproc = int(val)
         elif opt in ('--recurse', '-r'):
@@ -138,25 +136,25 @@ def main_bin():
     # expand directory if specified
     if recurse_dirs:
         expanded_files = []
-        for f in files:
+        for f in pyc_paths:
             if os.path.isdir(f):
                 for root, _, dir_files in os.walk(f):
                     for df in dir_files:
                         if df.endswith('.pyc') or df.endswith('.pyo'):
                             expanded_files.append(os.path.join(root, df))
-        files = expanded_files
+        pyc_paths = expanded_files
 
     # argl, commonprefix works on strings, not on path parts,
     # thus we must handle the case with files in 'some/classes'
     # and 'some/cmds'
-    src_base = os.path.commonprefix(files)
+    src_base = os.path.commonprefix(pyc_paths)
     if src_base[-1:] != os.sep:
         src_base = os.path.dirname(src_base)
     if src_base:
         sb_len = len( os.path.join(src_base, '') )
-        files = [f[sb_len:] for f in files]
+        pyc_paths = [f[sb_len:] for f in pyc_paths]
 
-    if not files:
+    if not pyc_paths:
         sys.stderr.write("No files given\n")
         usage()
 
@@ -164,7 +162,7 @@ def main_bin():
         outfile = None # use stdout
     elif outfile and os.path.isdir(outfile):
         out_base = outfile; outfile = None
-    elif outfile and len(files) > 1:
+    elif outfile and len(pyc_paths) > 1:
         out_base = outfile; outfile = None
 
     if timestamp:
@@ -172,10 +170,10 @@ def main_bin():
 
     if numproc <= 1:
         try:
-            result = main(src_base, out_base, files, codes, outfile,
+            result = main(src_base, out_base, pyc_paths, source_paths, outfile,
                           **options)
             result = list(result) + [options.get('do_verify', None)]
-            if len(files) > 1:
+            if len(pyc_paths) > 1:
                 mess = status_msg(do_verify, *result)
                 print('# ' + mess)
                 pass
@@ -191,8 +189,8 @@ def main_bin():
         except ImportError:
             from queue import Empty
 
-        fqueue = Queue(len(files)+numproc)
-        for f in files:
+        fqueue = Queue(len(pyc_paths)+numproc)
+        for f in pyc_paths:
             fqueue.put(f)
         for i in range(numproc):
             fqueue.put(None)
@@ -207,7 +205,7 @@ def main_bin():
                     if f is None:
                         break
                     (t, o, f, v) = \
-                      main(src_base, out_base, [f], codes, outfile, **options)
+                      main(src_base, out_base, [f], None, outfile, **options)
                     tot_files += t
                     okay_files += o
                     failed_files += f

uncompyle6/disas.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2015-2016, 2818 by Rocky Bernstein
+#  Copyright (c) 2015-2016, 2818-2019 by Rocky Bernstein
 #  Copyright (c) 2005 by Dan Pascu <dan@windowmaker.org>
 #  Copyright (c) 2000-2002 by hartmut Goebel <h.goebel@crazy-compilers.com>
 #  Copyright (c) 1999 John Aycock
@@ -73,18 +73,18 @@ def disco_loop(disasm, queue, real_out):
             pass
         pass
 
-def disassemble_fp(fp, outstream=None):
-    """
-    disassemble Python byte-code from an open file
-    """
-    (version, timestamp, magic_int, co, is_pypy,
-     source_size) = load_from_fp(fp)
-    if type(co) == list:
-        for con in co:
-            disco(version, con, outstream)
-    else:
-        disco(version, co, outstream, is_pypy=is_pypy)
-    co = None
+# def disassemble_fp(fp, outstream=None):
+#     """
+#     disassemble Python byte-code from an open file
+#     """
+#     (version, timestamp, magic_int, co, is_pypy,
+#      source_size) = load_from_fp(fp)
+#     if type(co) == list:
+#         for con in co:
+#             disco(version, con, outstream)
+#     else:
+#         disco(version, co, outstream, is_pypy=is_pypy)
+#     co = None
 
 def disassemble_file(filename, outstream=None):
     """
@@ -116,5 +116,6 @@ def _test():
         fn = sys.argv[1]
     disassemble_file(fn)
 
+
 if __name__ == "__main__":
     _test()

uncompyle6/main.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2018 Rocky Bernstein <rocky@gnu.org>
+# Copyright (C) 2018-2019 Rocky Bernstein <rocky@gnu.org>
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -12,10 +12,9 @@
 #
 #  You should have received a copy of the GNU General Public License
 #  along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
 import datetime, os, subprocess, sys
 
-from uncompyle6 import verify, IS_PYPY
+from uncompyle6 import verify, IS_PYPY, PYTHON_VERSION
 from xdis.code import iscode
 from xdis.magics import sysinfo2float
 from uncompyle6.disas import check_object_path
@@ -131,6 +130,22 @@ def decompile(
         # deparsing failed
         raise pysource.SourceWalkerError(str(e))
 
+def compile_file(source_path):
+    if source_path.endswith('.py'):
+        basename = source_path[:-3]
+    else:
+        basename = source_path
+
+    if hasattr(sys, 'pypy_version_info'):
+        bytecode_path = "%s-pypy%s.pyc" % (basename, PYTHON_VERSION)
+    else:
+        bytecode_path = "%s-%s.pyc" % (basename, PYTHON_VERSION)
+
+    print("compiling %s to %s" % (source_path, bytecode_path))
+    py_compile.compile(source_path, bytecode_path, 'exec')
+    return bytecode_path
+
+
 def decompile_file(filename, outstream=None, showasm=None, showast=False,
                    showgrammar=False, mapstream=None, do_fragments=False):
     """
@@ -162,14 +177,14 @@ def decompile_file(filename, outstream=None, showasm=None, showast=False,
 
 
 # FIXME: combine into an options parameter
-def main(in_base, out_base, files, codes, outfile=None,
+def main(in_base, out_base, compiled_files, source_files, outfile=None,
          showasm=None, showast=False, do_verify=False,
          showgrammar=False, raise_on_error=False,
          do_linemaps=False, do_fragments=False):
     """
     in_base	base directory for input files
     out_base	base directory for output files (ignored when
-    files	list of filenames to be uncompyled (relative to src_base)
+    files	list of filenames to be uncompyled (relative to in_base)
     outfile	write output to this filename (overwrites out_base)
 
     For redirecting output to
@@ -181,7 +196,10 @@ def main(in_base, out_base, files, codes, outfile=None,
     current_outfile = outfile
     linemap_stream = None
 
-    for filename in files:
+    for source_path in source_files:
+        compiled_files.append(compile_file(source_path))
+
+    for filename in compiled_files:
         infile = os.path.join(in_base, filename)
         # print("XXX", infile)
         if not os.path.exists(infile):
@@ -212,10 +230,11 @@ def main(in_base, out_base, files, codes, outfile=None,
                 else:
                     buffering = 0
                 sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', buffering)
-                tee = subprocess.Popen(["tee", current_outfile],
-                                       stdin=subprocess.PIPE)
-                os.dup2(tee.stdin.fileno(), sys.stdout.fileno())
-                os.dup2(tee.stdin.fileno(), sys.stderr.fileno())
+                if PYTHON_VERSION > 2.5:
+                    tee = subprocess.Popen(["tee", current_outfile],
+                                           stdin=subprocess.PIPE)
+                    os.dup2(tee.stdin.fileno(), sys.stdout.fileno())
+                    os.dup2(tee.stdin.fileno(), sys.stderr.fileno())
         else:
             if filename.endswith('.pyc'):
                 current_outfile = os.path.join(out_base, filename[0:-1])
@@ -321,10 +340,19 @@ def main(in_base, out_base, files, codes, outfile=None,
             sys.stdout.write("%s\r" %
                              status_msg(do_verify, tot_files, okay_files, failed_files,
                                         verify_failed_files, do_verify))
-            sys.stdout.flush()
+            try:
+                # FIXME: Something is weird with Pypy here
+                sys.stdout.flush()
+            except:
+                pass
     if current_outfile:
         sys.stdout.write("\n")
-        sys.stdout.flush()
+        try:
+            # FIXME: Something is weird with Pypy here
+            sys.stdout.flush()
+        except:
+            pass
+        pass
     return (tot_files, okay_files, failed_files, verify_failed_files)
 
 

uncompyle6/parser.py

@@ -35,7 +35,9 @@ class ParserError(Exception):
         return "Parse error at or near `%r' instruction at offset %s\n" % \
                (self.token, self.offset)
 
-nop_func = lambda self, args: None
+
+def nop_func(self, args):
+    return None
 
 class PythonParser(GenericASTBuilder):
 
@@ -743,6 +745,12 @@ def get_python_parser(
                 p = parse37.Python37Parser(debug_parser)
             else:
                 p = parse37.Python37ParserSingle(debug_parser)
+        elif version == 3.8:
+            import uncompyle6.parsers.parse38 as parse38
+            if compile_mode == 'exec':
+                p = parse38.Python38Parser(debug_parser)
+            else:
+                p = parse38.Python38ParserSingle(debug_parser)
         else:
             if compile_mode == 'exec':
                 p = parse3.Python3Parser(debug_parser)
@@ -790,6 +798,7 @@ def python_parser(version, co, out=sys.stdout, showasm=False,
     p = get_python_parser(version, parser_debug)
     return parse(p, tokens, customize)
 
+
 if __name__ == '__main__':
     def parse_test(co):
         from uncompyle6 import PYTHON_VERSION, IS_PYPY

uncompyle6/parsers/parse2.py

@@ -156,8 +156,10 @@ class Python2Parser(PythonParser):
         try_except      ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK
                             except_handler COME_FROM
 
+        # Note: except_stmts may have many jumps after END_FINALLY
         except_handler  ::= JUMP_FORWARD COME_FROM except_stmts
-                            END_FINALLY COME_FROM
+                            END_FINALLY come_froms
+
         except_handler  ::= jmp_abs COME_FROM except_stmts
                              END_FINALLY
 
@@ -463,7 +465,7 @@ class Python2Parser(PythonParser):
                         pass
                 self.add_unique_rules([
                     ('mkfunc ::= %s load_closure LOAD_CONST %s' %
-                     ('expr '* token.attr, opname))], customize)
+                     ('expr ' * token.attr, opname))], customize)
 
                 if self.version >= 2.7:
                     if i > 0:
@@ -529,14 +531,11 @@ class Python2Parser(PythonParser):
         # Dead code testing...
         # if lhs == 'while1elsestmt':
         #     from trepan.api import debug; debug()
-
         if lhs in ('aug_assign1', 'aug_assign2') and ast[0] and ast[0][0] in ('and', 'or'):
             return True
         elif lhs in ('raise_stmt1',):
-            # We will assme 'LOAD_ASSERT' will be handled by an assert grammar rule
-            return (tokens[first] == 'LOAD_ASSERT' and
-                    (last >= len(tokens) or tokens[last] not in
-                     ('COME_FROM', 'JUMP_BACK','JUMP_FORWARD')))
+            # We will assume 'LOAD_ASSERT' will be handled by an assert grammar rule
+            return (tokens[first] == 'LOAD_ASSERT' and (last >= len(tokens)))
         elif rule == ('or', ('expr', 'jmp_true', 'expr', '\\e_come_from_opt')):
             expr2 = ast[2]
             return expr2 == 'expr' and expr2[0] == 'LOAD_ASSERT'

uncompyle6/parsers/parse25.py

@@ -93,15 +93,17 @@ class Python25Parser(Python26Parser):
         super(Python25Parser, self).customize_grammar_rules(tokens, customize)
         if self.version == 2.5:
             self.check_reduce['try_except'] = 'tokens'
+        self.check_reduce['aug_assign1'] = 'AST'
 
-    ## Don't need this for 2.5 yet..
-    # def reduce_is_invalid(self, rule, ast, tokens, first, last):
-    #     invalid = super(Python25Parser,
-    #                     self).reduce_is_invalid(rule, ast,
-    #                                             tokens, first, last)
-    #     if invalid or tokens is None:
-    #         return invalid
-    #     return False
+    def reduce_is_invalid(self, rule, ast, tokens, first, last):
+        invalid = super(Python25Parser,
+                        self).reduce_is_invalid(rule, ast,
+                                                tokens, first, last)
+        if invalid or tokens is None:
+            return invalid
+        if rule == ('aug_assign1', ('expr', 'expr', 'inplace_op', 'store')):
+            return ast[0][0] == 'and'
+        return False
 
 
 class Python25ParserSingle(Python26Parser, PythonParserSingle):

uncompyle6/parsers/parse27.py

@@ -1,9 +1,10 @@
-#  Copyright (c) 2016-2018 Rocky Bernstein
+#  Copyright (c) 2016-2019 Rocky Bernstein
 #  Copyright (c) 2005 by Dan Pascu <dan@windowmaker.org>
 #  Copyright (c) 2000-2002 by hartmut Goebel <hartmut@goebel.noris.de>
 
 from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
-from uncompyle6.parser import PythonParserSingle
+from xdis import next_offset
+from uncompyle6.parser import PythonParserSingle, nop_func
 from uncompyle6.parsers.parse2 import Python2Parser
 
 class Python27Parser(Python2Parser):
@@ -155,7 +156,13 @@ class Python27Parser(Python2Parser):
 
         while1stmt        ::= SETUP_LOOP returns bp_come_from
         while1stmt        ::= SETUP_LOOP l_stmts_opt JUMP_BACK POP_BLOCK COME_FROM
+
         whilestmt         ::= SETUP_LOOP testexpr l_stmts_opt JUMP_BACK POP_BLOCK _come_froms
+
+        # Should this be JUMP_BACK+ ?
+        # JUMP_BACK should all be to the same location
+        whilestmt         ::= SETUP_LOOP testexpr l_stmts_opt JUMP_BACK JUMP_BACK POP_BLOCK _come_froms
+
         while1elsestmt    ::= SETUP_LOOP l_stmts JUMP_BACK POP_BLOCK
                               else_suitel COME_FROM
         whileelsestmt     ::= SETUP_LOOP testexpr l_stmts_opt JUMP_BACK POP_BLOCK
@@ -196,6 +203,13 @@ class Python27Parser(Python2Parser):
                            POP_BLOCK LOAD_CONST COME_FROM suite_stmts_opt
                            END_FINALLY
         """)
+        if 'PyPy' in customize:
+            # PyPy-specific customizations
+            self.addRule("""
+                        return_if_stmt ::= ret_expr RETURN_END_IF come_froms
+                        """, nop_func)
+
+
         super(Python27Parser, self).customize_grammar_rules(tokens, customize)
         self.check_reduce['and'] = 'AST'
         # self.check_reduce['or'] = 'AST'
@@ -203,14 +217,17 @@ class Python27Parser(Python2Parser):
         self.check_reduce['list_if_not'] = 'AST'
         self.check_reduce['list_if'] = 'AST'
         self.check_reduce['conditional_true'] = 'AST'
+        self.check_reduce['whilestmt'] = 'tokens'
         return
 
     def reduce_is_invalid(self, rule, ast, tokens, first, last):
         invalid = super(Python27Parser,
                         self).reduce_is_invalid(rule, ast,
                                                 tokens, first, last)
+
         if invalid:
             return invalid
+
         if rule == ('and', ('expr', 'jmp_false', 'expr', '\\e_come_from_opt')):
             # Test that jmp_false jumps to the end of "and"
             # or that it jumps to the same place as the end of "and"
@@ -220,6 +237,12 @@ class Python27Parser(Python2Parser):
                         tokens[last].pattr == jmp_false.pattr)
         elif rule[0] == ('raise_stmt1'):
             return ast[0] == 'expr' and ast[0][0] == 'or'
+        elif rule[0] in ('assert', 'assert2'):
+            jump_inst = ast[1][0]
+            jump_target = jump_inst.attr
+            return not (last >= len(tokens)
+                        or jump_target == tokens[last].offset
+                        or jump_target == next_offset(ast[-1].op, ast[-1].opc, ast[-1].offset))
         elif rule == ('list_if_not', ('expr', 'jmp_true', 'list_iter')):
             jump_inst = ast[1][0]
             jump_offset = jump_inst.attr
@@ -235,6 +258,16 @@ class Python27Parser(Python2Parser):
             jmp_target = jmp_true.offset + jmp_true.attr + 3
             return not (jmp_target == tokens[last].offset or
                         tokens[last].pattr == jmp_true.pattr)
+
+        elif (rule[0] == 'whilestmt' and
+              rule[1][0:-2] ==
+                      ('SETUP_LOOP', 'testexpr', 'l_stmts_opt',
+                       'JUMP_BACK', 'JUMP_BACK')):
+            # Make sure that the jump backs all go to the same place
+            i = last-1
+            while (tokens[i] != 'JUMP_BACK'):
+                i -= 1
+            return tokens[i].attr != tokens[i-1].attr
         # elif rule[0] == ('conditional_true'):
         #     # FIXME: the below is a hack: we check expr for
         #     # nodes that could have possibly been a been a Boolean.
@@ -263,7 +296,7 @@ if __name__ == '__main__':
             """.split()))
         remain_tokens = set(tokens) - opcode_set
         import re
-        remain_tokens = set([re.sub('_\d+$', '', t)
+        remain_tokens = set([re.sub(r'_\d+$', '', t)
                              for t in remain_tokens])
         remain_tokens = set([re.sub('_CONT$', '', t)
                              for t in remain_tokens])