Merge branch 'python-3.3-to-3.5' into python-2.4

HOW-TO-REPORT-A-BUG.md

@@ -1,21 +1,54 @@
 <!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc -->
 **Table of Contents**
 
-- [The difficulty of the problem](#the-difficulty-of-the-problem)
+- [Ethics](#ethics)
+- [The importance of your bug report](#the-importance-of-your-bug-report)
+- [The difficulty of the problem and your bug](#the-difficulty-of-the-problem-and-your-bug)
 - [Is it really a bug?](#is-it-really-a-bug)
     - [Do you have valid bytecode?](#do-you-have-valid-bytecode)
     - [Semantic equivalence vs. exact source code](#semantic-equivalence-vs-exact-source-code)
 - [What to send (minimum requirements)](#what-to-send-minimum-requirements)
 - [What to send (additional helpful information)](#what-to-send-additional-helpful-information)
     - [But I don't *have* the source code!](#but-i-dont-have-the-source-code)
-    - [But I don't *have* the source code and am incapable of figuring how how to do a hand disassembly!](#but-i-dont-have-the-source-code-and-am-incapable-of-figuring-how-how-to-do-a-hand-disassembly)
+        - [But I don't *have* the source code and am incapable of figuring how to do a hand disassembly!](#but-i-dont-have-the-source-code-and-am-incapable-of-figuring-how-to-do-a-hand-disassembly)
 - [Narrowing the problem](#narrowing-the-problem)
 - [Karma](#karma)
 - [Confidentiality of Bug Reports](#confidentiality-of-bug-reports)
-- [Ethics](#ethics)
 
 <!-- markdown-toc end -->
-# The difficulty of the problem
+
+TL;DR (too long; didn't read)
+
+* Don't do something illegal. And don't ask me to do something illegal or help you do something illegal
+* We already have an infinite supply of decompilation bugs that need fixing, and an automated mechanism for finding more. Decompilation bugs get addressed by easiness to fix and by whim. If you expect yours to be fixed ahead of those, you need to justify why.
+* When asking for help, you may be asked for what you've tried on your own first. There are plenty of sources of information about this code.
+* If you are looking for *timely* help or support, well, that is typically known paid service. I don't really have a mechanism for that since I have a full-time job. But supporting the project is an approximation.
+* Submitting a bug or issue report that is likely to get acted upon may require a bit of effort on your part to make it easy for the problem solver. If you are not willing to do that, please don't waste our time. As indicated above, supporting the project will increase the likelihood of your issue getting noticed and acted upon.
+
+# Ethics
+
+I do not condone using this program for unethical or illegal purposes. More detestable, at least to me, is asking for help to assist you in something that might not legitimate.
+
+Don't use the issue tracker for such solicitations. To try to stave off illegitimate behavior, you should note that the issue tracker, the code, and bugs mentioned in that are in the open: there is no
+confidentiality. You may be asked about the authorship or claimed ownership of the bytecode. If I think something is not quite right, I may label the issue questionable which may make the it easier those who are looking for illegal activity.
+
+
+# The importance of your bug report
+
+For many open-source projects bugs where the expectation is that bugs are rare, reporting bugs in a *thoughtful* way can be helpful. See also [How to Ask Questions the Smart Way](https://www.osadl.org/Eric-Raymond-How-to-ask-questions-the-s.questions-the-smart-way.0.html).
+
+In this project though, most of the bug reports boil down to the something like: I have I am trying to reverse engineer some code that I am not the author/owner and that person doesn't want me to have access to. I am hitting a problem somewhere along the line which might have to do with decompilation, but it could be something else like how the bytecode was extracted, some problem in deliberately obfuscated code, or the use some kind of Python bytecode version that isn't supported by the decompiler.
+
+While you are free to report these, unless you sponsor the project, I may close them with about the same amount of effort spent that I think was used to open the report for them. And if you spent a considerable amount of time to create the bug report but didn't follow instructions given here and in the issue template, I am sorry in advance. Just go back, read, and follow instructions.
+
+This project already has an infinite supply of bugs that have been narrowed to the most minimal form and where I have source code to compare against. And in the unlikely event this supply runs out, I have automated means for generating *another* infinite supply.
+
+In this project the task of justifying why addressing your bug is of use to the community, and why it should be prioritized over the others, is the bug reporter's responsibility.
+
+While in the abstract, I have no problem answering questions about how to read a Python traceback or install Python software, or trying to understand what is going wrong in your particular setup, I am not a paid support person and there other things I'd rather be doing with my limited volunteer time. So save us both time, effort, and aggravation: use other avenues like StackOverflow. Again, justifying why you should receive unpaid help is the help requester's responsibility.
+
+
+# The difficulty of the problem and your bug
 
 This decompiler is a constant work in progress: Python keeps
 changing, and so does its code generation.
@@ -27,7 +60,7 @@ But at any given time, there are a number of valid Python bytecode files that I
 
 There are far more bug reporters than there are bug fixers.
 
-Unless you are a sponsor of this project, it may take a while, maybe a week or so, before the bug report is noticed, let alone acted upon. Things eventually get fixed, but it may take years.
+Unless you are a sponsor of this project, it may take a while, maybe a week or so, before the bug report is noticed, let alone acted upon. Things eventually get fixed, but it may take years. And if your bug hasn't been narrowed, it might happen as a result of some other bug fix.
 
 # Is it really a bug?
 
@@ -35,7 +68,7 @@ Unless you are a sponsor of this project, it may take a while, maybe a week or s
 ## Do you have valid bytecode?
 
 As mentioned in README.rst, this project doesn't handle obfuscated
-code or release candidates. See README.rst for suggestions for how to remove some kinds of
+code, release candidates, and the most recent versions of Python: version 3.9 and up. See README.rst for suggestions for how to remove some kinds of
 obfuscation.
 
 Checking if bytecode is valid is pretty simple: disassemble the code.
@@ -45,18 +78,7 @@ disassembler called `pydisasm`.
 
 ## Semantic equivalence vs. exact source code
 
-Consider how Python compiles something like "(x*y) + 5". Early on
+Consider how Python compiles something like "(x*y) + 5". Early on Python creates an "abstract syntax tree" (AST) for this. And this is "abstract" in the sense that unimportant, redundant or unnecessary items have been removed. Here, this means that any notion that you wrote "x+y" in parenthesis is lost, since in this context they are unneeded. Also lost is the fact that the multiplication didn't have spaces around it while the addition did. It should not come as a surprise then that the bytecode which is derived from the AST also has no notion of such possible variation. Generally this kind of thing isn't noticed since the Python community has laid out a very rigid set of formatting guidelines; and it has largely beaten the community into compliance.
-Python creates an "abstract syntax tree" (AST) for this. And this is
-"abstract" in the sense that unimportant, redundant or unnecessary
-items have been removed. Here, this means that any notion that you
-wrote "x+y" in parenthesis is lost, since in this context they are
-unneeded. Also lost is the fact that the multiplication didn't have
-spaces around it while the addition did. It should not come as a
-surprise then that the bytecode which is derived from the AST also has
-no notion of such possible variation. Generally this kind of thing
-isn't noticed since the Python community has laid out a very rigid set
-of formatting guidelines; and it has largely beaten the community into
-compliance.
 
 Almost all versions of Python can perform some sort of code
 improvement that can't be undone. In earlier versions of Python it is
@@ -138,7 +160,7 @@ Also try to narrow the bug. See below.
 Some kind folks also give the invocation they used and the output
 which usually includes an error message produced. This is
 helpful. From this, I can figure out what OS you are running this on
-and what version of *uncomplye6* was used. Therefore, if you _don't_
+and what version of *uncompyle6* was used. Therefore, if you _don't_
 provide the input command and the output from that, please give:
 
 * _uncompyle6_ version used
@@ -155,7 +177,7 @@ There is Python assembly code on parse errors, so simply by hand decompile that.
 Well, you could learn. No one is born into this world knowing how to disassemble Python bytecode. And as Richard Feynman once said, "What one fool can learn, so can another."
 
 If this is too difficult, or too time consuming, or not of interest to you, then you might consider [sponsoring](https://github.com/sponsors/rocky) the project. [Crazy
-Compilers](http://www.crazy-compilers.com/decompyle/) offers a byte-code decompiler service for versions of Python up to 2.6. (If there are others around let me know and I'll list them here.)
+Compilers](http://www.crazy-compilers.com/decompyle/) offers a byte-code decompiler service for versions of Python up to 2.6. (If there are others around let me know and I'll list them here.) Don't be surprised if I ask you to pay for work (if I think the work is ethical) when you want me to work on your problem that I think isn't of interest or benefit to anyone but yourself or a small limited number of people, or I think the need is questionable.
 
 # Narrowing the problem
 
@@ -177,8 +199,11 @@ likely the problem will be fixed and fixed sooner.
 # Karma
 
 I realize that following the instructions given herein puts a bit of
-burden on the bug reporter. This is justified as attempts to balance somewhat the burden and effort needed to fix the bug and the attempts to balance number of would-be bug reporters with the number of bug fixers. Better bug reporters are more likely to move
+burden on the bug reporter. This is justified since it attempts to balance
-in the category of bug fixers.
+the burden and effort needed to fix the bug with the amount of effort to report the problem. And it attempts
+to balance number of would-be bug reporters with the number of bug
+fixers. Better bug reporters are more likely to move in the category
+of bug fixers.
 
 The barrier to reporting a big is pretty small: all you really need is
 a github account, and the ability to type something after clicking
@@ -189,9 +214,13 @@ That said, bugs sometimes get fixed even though these instructions are not follo
 
 I may take into consideration is the bug reporter's karma.
 
-* Have you demonstrably contributed to open source? I may look at your  github profile to see what contributions you have made, how popular those contributions are, or how popular you are.
+* Have you demonstrably contributed to open source? I may look at your github profile to see what contributions you have made, how popular those contributions are, or how popular you are.
-* How appreciative are you? Have you starred this project that you are   seeking help from? Have you starred _any_ github project? And the above  two kind of feed into ...
+* How appreciative are you? Have you starred this project that you are seeking help from? Have you starred _any_ github project? And the above two kind of feed into ...
-* Attitude. Some people feel that they are doing me and the world a great favor by just pointing out that there is a problem whose solution would greatly benefit them. (This might account partially those that have this attitude often don't read or follow instructions such as those given here.)
+* Attitude. Some people feel that they are doing me and the world a
+  great favor by just pointing out that there is a problem whose
+  solution would greatly benefit them. (This might account partially
+  for the fact that those that have this attitude often don't read or
+  follow instructions such as those given here.)
 
 
 # Confidentiality of Bug Reports
@@ -205,17 +234,3 @@ However feel free to remove any comments, and modify variable names
 or constants in the source code.
 
 If there is some legitimate reason to keep confidentiality, you can contact me by email to explain the extenuating circumstances. However I tend to discard without reading anonymous email.
-
-# Ethics
-
-I do not condone using this program for unethical or illegal purposes.
-More detestable, at least to me, is asking for help to assist you in
-something that might not legitimate.
-
-Don't use the issue tracker for such solicitations. To try to stave
-off illegitimate behavior, you should note that the issue tracker, the
-code, and bugs mentioned in that are in the open: there is no
-confidentiality. You may be asked about the authorship or claimed
-ownership of the bytecode. If I think something is not quite right, I
-may label the issue questionable which may make the it easier those
-who are looking for illegal activity.

README.rst

@@ -2,6 +2,8 @@
 
 |packagestatus|
 
+.. contents::
+
 uncompyle6
 ==========
 
@@ -80,7 +82,7 @@ latest Python version, with the exception of Python 3.0 through
 3.2. Volunteers are welcome to address these deficiencies if there a
 desire to do so.
 
-nThe way it does this though is by segregating consecutive Python versions into
+The way it does this though is by segregating consecutive Python versions into
 git branches:
 
 master
@@ -99,7 +101,12 @@ versions.
 Installation
 ------------
 
-This uses setup.py, so it follows the standard Python routine::
+You can install from PyPI using the name ``uncompyle6``::
+
+    pip install uncompyle6
+
+
+To install from source code, this project uses setup.py, so it follows the standard Python routine::
 
     $ pip install -e .  # set up to run from source tree
 
@@ -144,7 +151,7 @@ Verification
 
 In older versions of Python it was possible to verify bytecode by
 decompiling bytecode, and then compiling using the Python interpreter
-for that bytecode version. Having done this the bytecode produced
+for that bytecode version. Having done this, the bytecode produced
 could be compared with the original bytecode. However as Python's code
 generation got better, this no longer was feasible.
 
@@ -158,7 +165,7 @@ You can also cross compare the results with either another version of
 `uncompyle6` since there are are sometimes regressions in decompiling
 specific bytecode as the overall quality improves.
 
-For Python 3.7 and above, the code in decompyle3_ is generally
+For Python 3.7 and 3.8, the code in decompyle3_ is generally
 better.
 
 Or try specific another python decompiler like uncompyle2_, unpyc37_,
@@ -227,12 +234,14 @@ handled.
 
 We also don't handle PJOrion_ or otherwise obfuscated code. For
 PJOrion try: PJOrion Deobfuscator_ to unscramble the bytecode to get
-valid bytecode before trying this tool. This program can't decompile
+valid bytecode before trying this tool; pydecipher_ might help with that.
-Microsoft Windows EXE files created by Py2EXE_, although we can
+
-probably decompile the code after you extract the bytecode
+This program can't decompile Microsoft Windows EXE files created by
-properly. Handling pathologically long lists of expressions or
+Py2EXE_, although we can probably decompile the code after you extract
-statements is slow. We don't handle Cython_ or MicroPython which don't
+the bytecode properly. `Pydeinstaller <https://github.com/charles-dyfis-net/pydeinstaller>`_ may help with unpacking Pyinstaller bundlers.
-use bytecode.
+
+Handling pathologically long lists of expressions or statements is
+slow. We don't handle Cython_ or MicroPython which don't use bytecode.
 
 There are numerous bugs in decompilation. And that's true for every
 other CPython decompiler I have encountered, even the ones that
@@ -246,7 +255,7 @@ because it is harder to do so. The good news, at least from my
 standpoint, is that I think I understand what's needed to address the
 problems in a more robust way. But right now until such time as
 project is better funded, I do not intend to make any serious effort
-to support Python versions 3.8 and above here, including bugs that might come
+to support Python versions 3.8 or 3.9, including bugs that might come
 in. I imagine at some point I may be interested in it.
 
 You can easily find bugs by running the tests against the standard
@@ -270,21 +279,21 @@ issues above the queue of other things I might be doing instead.
 See Also
 --------
 
-* https://github.com/rocky/python-decompile3 : Much smaller and more modern code, focusing on 3.7+. Changes in that will get migrated back here.
+* https://github.com/rocky/python-decompile3 : Much smaller and more modern code, focusing on 3.7 and 3.8. Changes in that will get migrated back here.
 * https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique than what is used here. Currently unmaintained.
 * https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Includes some fixes like supporting function annotations. Currently unmaintained.
-* https://github.com/wibiti/uncompyle2 : supports Python 2.7 only, but does that fairly well. There are situations where :code:`uncompyle6` results are incorrect while :code:`uncompyle2` results are not, but more often uncompyle6 is correct when uncompyle2 is not. Because :code:`uncompyle6` adheres to accuracy over idiomatic Python, :code:`uncompyle2` can produce more natural-looking code when it is correct. Currently :code:`uncompyle2` is lightly maintained. See its issue `tracker <https://github.com/wibiti/uncompyle2/issues>`_ for more details
+* https://github.com/wibiti/uncompyle2 : supports Python 2.7 only, but does that fairly well. There are situations where :code:`uncompyle6` results are incorrect while :code:`uncompyle2` results are not, but more often uncompyle6 is correct when uncompyle2 is not. Because :code:`uncompyle6` adheres to accuracy over idiomatic Python, :code:`uncompyle2` can produce more natural-looking code when it is correct. Currently :code:`uncompyle2` is lightly maintained. See its issue `tracker <https://github.com/wibiti/uncompyle2/issues>`_ for more details.
 * `How to report a bug <https://github.com/rocky/python-uncompyle6/blob/master/HOW-TO-REPORT-A-BUG.md>`_
 * The HISTORY_ file.
 * https://github.com/rocky/python-xdis : Cross Python version disassembler
 * https://github.com/rocky/python-xasm : Cross Python version assembler
 * https://github.com/rocky/python-uncompyle6/wiki : Wiki Documents which describe the code and aspects of it in more detail
-* https://github.com/zrax/pycdc : The README for this C++ code says it aims to support all versions of Python. It is best for Python versions around 2.7 and 3.3 when the code was initially developed. Accuracy for current versions of Python3 and early versions of Python is lacking. Without major effort, it is unlikely it can be made to support current Python 3. See its `issue tracker <https://github.com/zrax/pycdc/issues>`_ for details. Currently lightly maintained.
+* https://github.com/zrax/pycdc : The README for this C++ code says it aims to support all versions of Python. You can aim your slign shot for the moon too, but I doubt you are going to hit it. This code is best for Python versions around 2.7 and 3.3 when the code was initially developed. Accuracy for current versions of Python3 and early versions of Python is lacking. Without major effort, it is unlikely it can be made to support current Python 3. See its `issue tracker <https://github.com/zrax/pycdc/issues>`_ for details. Currently lightly maintained.
 
 
 .. _Cython: https://en.wikipedia.org/wiki/Cython
 .. _trepan: https://pypi.python.org/pypi/trepan3k
-.. _compiler: https://pypi.python.org/pypi/spark_parser
+.. _compiler: https://github.com/rocky/python-uncompyle6/wiki/How-does-this-code-work%3F
 .. _HISTORY: https://github.com/rocky/python-uncompyle6/blob/master/HISTORY.md
 .. _report_bug: https://github.com/rocky/python-uncompyle6/blob/master/HOW-TO-REPORT-A-BUG.md
 .. _debuggers: https://pypi.python.org/pypi/trepan3k
@@ -297,6 +306,7 @@ See Also
 .. |buildstatus| image:: https://travis-ci.org/rocky/python-uncompyle6.svg :target: https://travis-ci.org/rocky/python-uncompyle6
 .. |packagestatus| image:: https://repology.org/badge/vertical-allrepos/python:uncompyle6.svg :target: https://repology.org/project/python:uncompyle6/versions
 .. _PJOrion: http://www.koreanrandom.com/forum/topic/15280-pjorion-%D1%80%D0%B5%D0%B4%D0%B0%D0%BA%D1%82%D0%B8%D1%80%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5-%D0%BA%D0%BE%D0%BC%D0%BF%D0%B8%D0%BB%D1%8F%D1%86%D0%B8%D1%8F-%D0%B4%D0%B5%D0%BA%D0%BE%D0%BC%D0%BF%D0%B8%D0%BB%D1%8F%D1%86%D0%B8%D1%8F-%D0%BE%D0%B1%D1%84
+.. _pydecipher: https://github.com/mitre/pydecipher
 .. _Deobfuscator: https://github.com/extremecoders-re/PjOrion-Deobfuscator
 .. _Py2EXE: https://en.wikipedia.org/wiki/Py2exe
 .. |Supported Python Versions| image:: https://img.shields.io/pypi/pyversions/uncompyle6.svg

uncompyle6/__init__.py

@@ -1,5 +1,5 @@
 """
-  Copyright (c) 2015, 2018, 2021 by Rocky Bernstein
+  Copyright (c) 2015, 2018, 2021-2022 by Rocky Bernstein
   Copyright (c) 2000 by hartmut Goebel <h.goebel@crazy-compilers.com>
   Copyright (c) 1999 John Aycock
 
@@ -36,8 +36,7 @@ if hasattr(sys, "setrecursionlimit"):
     # pyston doesn't have setrecursionlimit
     sys.setrecursionlimit(5000)
 
-import uncompyle6.semantics.pysource
+from uncompyle6.semantics import fragments, pysource
-import uncompyle6.semantics.fragments
 
 # Export some functions
 from uncompyle6.main import decompile_file  # noqa
@@ -45,5 +44,13 @@ from uncompyle6.main import decompile_file  # noqa
 # Convenience functions so you can say:
 # from uncompyle6 import (code_deparse, deparse_code2str)
 
-deparse_code2str = uncompyle6.semantics.pysource.deparse_code2str
+from uncompyle6.semantics.pysource import code_deparse, deparse_code2str
-code_deparse = uncompyle6.semantics.pysource.code_deparse
+
+__all__ = [
+    "__version__",
+    "code_deparse",
+    "decompile_file",
+    "deparse_code2str",
+    "fragments",
+    "pysource",
+]

uncompyle6/main.py

@@ -12,15 +12,15 @@
 #
 #  You should have received a copy of the GNU General Public License
 #  along with this program.  If not, see <http://www.gnu.org/licenses/>.
-import datetime, os, subprocess, sys
+import datetime, os, py_compile, subprocess, sys
 
-from uncompyle6 import verify
 from xdis import iscode
 from xdis.version_info import IS_PYPY, PYTHON_VERSION_TRIPLE, version_tuple_to_str
 from uncompyle6.disas import check_object_path
 from uncompyle6.semantics import pysource
 from uncompyle6.semantics.pysource import PARSER_DEFAULT_DEBUG
 from uncompyle6.parser import ParserError
+from uncompyle6 import verify
 from uncompyle6.version import __version__
 
 # from uncompyle6.linenumbers import line_number_mapping
@@ -401,7 +401,6 @@ def main(
         else:  # uncompile successful
             if current_outfile:
                 outstream.close()
-
                 if do_verify:
                     try:
                         msg = verify.compare_code_with_srcfile(
@@ -457,7 +456,6 @@ def main(
                         okay_files,
                         failed_files,
                         verify_failed_files,
-                        do_verify,
                     ),
                 )
             )
@@ -494,20 +492,12 @@ else:
         return ""
 
 
-def status_msg(
+def status_msg(do_verify, tot_files, okay_files, failed_files, verify_failed_files):
-    do_verify, tot_files, okay_files, failed_files, verify_failed_files, weak_verify
-):
-    if weak_verify == "weak":
-        verification_type = "weak "
-    elif weak_verify == "verify-run":
-        verification_type = "run "
-    else:
-        verification_type = ""
     if tot_files == 1:
         if failed_files:
             return "\n# decompile failed"
         elif verify_failed_files:
-            return "\n# decompile %sverification failed" % verification_type
+            return "\n# decompile run verification failed"
         else:
             return "\n# Successfully decompiled file"
             pass
@@ -517,6 +507,4 @@ def status_msg(
         okay_files,
         failed_files,
     )
-    if do_verify:
-        mess += ", %i %sverification failed" % (verify_failed_files, verification_type)
     return mess

uncompyle6/semantics/customize3.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2018-2021 by Rocky Bernstein
+#  Copyright (c) 2018-2022 by Rocky Bernstein
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by

uncompyle6/semantics/fragments.py

@@ -2288,7 +2288,7 @@ def deparsed_find(tup, deparsed, code):
 #     def test():
 #         import os, sys
 
-#     def get_dups(li: list) -> set:
+#     def get_dups(li: list):
 #         dups = {}
 #         for item in li:
 #             dups[item] = dups.get(item, -1) + 1

uncompyle6/semantics/pysource.py

@@ -196,6 +196,15 @@ PARSER_DEFAULT_DEBUG = {
     "dups": False,
 }
 
+PARSER_DEFAULT_DEBUG = {
+    "rules": False,
+    "transition": False,
+    "reduce": False,
+    "errorstack": "full",
+    "context": True,
+    "dups": False,
+}
+
 TREE_DEFAULT_DEBUG = {"before": False, "after": False}
 
 DEFAULT_DEBUG_OPTS = {
@@ -204,6 +213,7 @@ DEFAULT_DEBUG_OPTS = {
     "grammar": dict(PARSER_DEFAULT_DEBUG),
 }
 
+
 class SourceWalkerError(Exception):
     def __init__(self, errmsg):
         self.errmsg = errmsg
@@ -1462,7 +1472,10 @@ class SourceWalker(GenericASTTraversal, object):
             tree in ("stmt", "sstmt", "return", "return_expr", "return_expr_lambda")
         ):
             self.prec = 100
-            tree = tree[0]
+            if tree[0] in ("dom_start", "dom_start_opt"):
+                tree = tree[1]
+            else:
+                tree = tree[0]
         return tree
 
     def closure_walk(self, node, collection_index):