Merge hell

This commit is contained in:
rocky
2024-05-30 04:47:35 -04:00
parent 6b0c909460
commit 3939466281
3 changed files with 116 additions and 145 deletions

158
PKG-INFO
View File

@@ -7,54 +7,54 @@ Author: Rocky Bernstein, Hartmut Goebel, John Aycock, and others
Author-email: rb@dustyfeet.com Author-email: rb@dustyfeet.com
License: GPL3 License: GPL3
Description: |buildstatus| |Pypi Installs| |Latest Version| |Supported Python Versions| Description: |buildstatus| |Pypi Installs| |Latest Version| |Supported Python Versions|
|packagestatus| |packagestatus|
.. contents:: .. contents::
uncompyle6 uncompyle6
========== ==========
A native Python cross-version decompiler and fragment decompiler. A native Python cross-version decompiler and fragment decompiler.
The successor to decompyle, uncompyle, and uncompyle2. The successor to decompyle, uncompyle, and uncompyle2.
Introduction Introduction
------------ ------------
*uncompyle6* translates Python bytecode back into equivalent Python *uncompyle6* translates Python bytecode back into equivalent Python
source code. It accepts bytecodes from Python version 1.0 to version source code. It accepts bytecodes from Python version 1.0 to version
3.8, spanning over 24 years of Python releases. We include Dropbox's 3.8, spanning over 24 years of Python releases. We include Dropbox's
Python 2.5 bytecode and some PyPy bytecodes. Python 2.5 bytecode and some PyPy bytecodes.
Why this? Why this?
--------- ---------
Ok, I'll say it: this software is amazing. It is more than your Ok, I'll say it: this software is amazing. It is more than your
normal hacky decompiler. Using compiler_ technology, the program normal hacky decompiler. Using compiler_ technology, the program
creates a parse tree of the program from the instructions; nodes at creates a parse tree of the program from the instructions; nodes at
the upper levels that look a little like what might come from a Python the upper levels that look a little like what might come from a Python
AST. So we can really classify and understand what's going on in AST. So we can really classify and understand what's going on in
sections of Python bytecode. sections of Python bytecode.
Building on this, another thing that makes this different from other Building on this, another thing that makes this different from other
CPython bytecode decompilers is the ability to deparse just CPython bytecode decompilers is the ability to deparse just
*fragments* of source code and give source-code information around a *fragments* of source code and give source-code information around a
given bytecode offset. given bytecode offset.
I use the tree fragments to deparse fragments of code *at run time* I use the tree fragments to deparse fragments of code *at run time*
inside my trepan_ debuggers_. For that, bytecode offsets are recorded inside my trepan_ debuggers_. For that, bytecode offsets are recorded
and associated with fragments of the source code. This purpose, and associated with fragments of the source code. This purpose,
although compatible with the original intention, is yet a little bit although compatible with the original intention, is yet a little bit
different. See this_ for more information. different. See this_ for more information.
Python fragment deparsing given an instruction offset is useful in Python fragment deparsing given an instruction offset is useful in
showing stack traces and can be incorporated into any program that showing stack traces and can be incorporated into any program that
wants to show a location in more detail than just a line number at wants to show a location in more detail than just a line number at
runtime. This code can be also used when source-code information does runtime. This code can be also used when source-code information does
not exist and there is just bytecode. Again, my debuggers make use of not exist and there is just bytecode. Again, my debuggers make use of
this. this.
There were (and still are) a number of decompyle, uncompyle, There were (and still are) a number of decompyle, uncompyle,
uncompyle2, uncompyle3 forks around. Many of them come basically from uncompyle2, uncompyle3 forks around. Many of them come basically from
the same code base, and (almost?) all of them are no longer actively the same code base, and (almost?) all of them are no longer actively
@@ -65,148 +65,148 @@ Description: |buildstatus| |Pypi Installs| |Latest Version| |Supported Python V
forward*. There is some serious refactoring and cleanup in this code forward*. There is some serious refactoring and cleanup in this code
base over those old forks. Even more experimental refactoring is going base over those old forks. Even more experimental refactoring is going
on in decompyle3_. on in decompyle3_.
This demonstrably does the best in decompiling Python across all This demonstrably does the best in decompiling Python across all
Python versions. And even when there is another project that only Python versions. And even when there is another project that only
provides decompilation for subset of Python versions, we generally do provides decompilation for subset of Python versions, we generally do
demonstrably better for those as well. demonstrably better for those as well.
How can we tell? By taking Python bytecode that comes distributed with How can we tell? By taking Python bytecode that comes distributed with
that version of Python and decompiling these. Among those that that version of Python and decompiling these. Among those that
successfully decompile, we can then make sure the resulting programs successfully decompile, we can then make sure the resulting programs
are syntactically correct by running the Python interpreter for that are syntactically correct by running the Python interpreter for that
bytecode version. Finally, in cases where the program has a test for bytecode version. Finally, in cases where the program has a test for
itself, we can run the check on the decompiled code. itself, we can run the check on the decompiled code.
We use an automated processes to find bugs. In the issue trackers for We use an automated processes to find bugs. In the issue trackers for
other decompilers, you will find a number of bugs we've found along other decompilers, you will find a number of bugs we've found along
the way. Very few to none of them are fixed in the other decompilers. the way. Very few to none of them are fixed in the other decompilers.
Requirements Requirements
------------ ------------
The code in the git repository can be run from Python 2.4 to the The code in the git repository can be run from Python 2.4 to the
latest Python version, with the exception of Python 3.0 through latest Python version, with the exception of Python 3.0 through
3.2. Volunteers are welcome to address these deficiencies if there a 3.2. Volunteers are welcome to address these deficiencies if there a
desire to do so. desire to do so.
The way it does this though is by segregating consecutive Python versions into The way it does this though is by segregating consecutive Python versions into
git branches: git branches:
master master
Python 3.6 and up (uses type annotations) Python 3.6 and up (uses type annotations)
python-3.3-to-3.5 python-3.3-to-3.5
Python 3.3 through 3.5 (Generic Python 3) Python 3.3 through 3.5 (Generic Python 3)
python-2.4 python-2.4
Python 2.4 through 2.7 (Generic Python 2) Python 2.4 through 2.7 (Generic Python 2)
PyPy 3-2.4 and later works as well. PyPy 3-2.4 and later works as well.
The bytecode files it can read have been tested on Python The bytecode files it can read have been tested on Python
bytecodes from versions 1.4, 2.1-2.7, and 3.0-3.8 and later PyPy bytecodes from versions 1.4, 2.1-2.7, and 3.0-3.8 and later PyPy
versions. versions.
Installation Installation
------------ ------------
You can install from PyPI using the name ``uncompyle6``:: You can install from PyPI using the name ``uncompyle6``::
pip install uncompyle6 pip install uncompyle6
To install from source code, this project uses setup.py, so it follows the standard Python routine:: To install from source code, this project uses setup.py, so it follows the standard Python routine::
$ pip install -e . # set up to run from source tree $ pip install -e . # set up to run from source tree
or:: or::
$ python setup.py install # may need sudo $ python setup.py install # may need sudo
A GNU Makefile is also provided so :code:`make install` (possibly as root or A GNU Makefile is also provided so :code:`make install` (possibly as root or
sudo) will do the steps above. sudo) will do the steps above.
Running Tests Running Tests
------------- -------------
:: ::
make check make check
A GNU makefile has been added to smooth over setting running the right A GNU makefile has been added to smooth over setting running the right
command, and running tests from fastest to slowest. command, and running tests from fastest to slowest.
If you have remake_ installed, you can see the list of all tasks If you have remake_ installed, you can see the list of all tasks
including tests via :code:`remake --tasks` including tests via :code:`remake --tasks`
Usage Usage
----- -----
Run Run
:: ::
$ uncompyle6 *compiled-python-file-pyc-or-pyo* $ uncompyle6 *compiled-python-file-pyc-or-pyo*
For usage help: For usage help:
:: ::
$ uncompyle6 -h $ uncompyle6 -h
Verification Verification
------------ ------------
In older versions of Python it was possible to verify bytecode by In older versions of Python it was possible to verify bytecode by
decompiling bytecode, and then compiling using the Python interpreter decompiling bytecode, and then compiling using the Python interpreter
for that bytecode version. Having done this, the bytecode produced for that bytecode version. Having done this, the bytecode produced
could be compared with the original bytecode. However as Python's code could be compared with the original bytecode. However as Python's code
generation got better, this no longer was feasible. generation got better, this no longer was feasible.
If you want Python syntax verification of the correctness of the If you want Python syntax verification of the correctness of the
decompilation process, add the :code:`--syntax-verify` option. However since decompilation process, add the :code:`--syntax-verify` option. However since
Python syntax changes, you should use this option if the bytecode is Python syntax changes, you should use this option if the bytecode is
the right bytecode for the Python interpreter that will be checking the right bytecode for the Python interpreter that will be checking
the syntax. the syntax.
You can also cross compare the results with another version of You can also cross compare the results with another version of
`uncompyle6` since there are sometimes regressions in decompiling `uncompyle6` since there are sometimes regressions in decompiling
specific bytecode as the overall quality improves. specific bytecode as the overall quality improves.
For Python 3.7 and 3.8, the code in decompyle3_ is generally For Python 3.7 and 3.8, the code in decompyle3_ is generally
better. better.
Or try specific another python decompiler like uncompyle2_, unpyc37_, Or try specific another python decompiler like uncompyle2_, unpyc37_,
or pycdc_. Since the later two work differently, bugs here often or pycdc_. Since the later two work differently, bugs here often
aren't in that, and vice versa. aren't in that, and vice versa.
There is an interesting class of these programs that is readily There is an interesting class of these programs that is readily
available give stronger verification: those programs that when run available give stronger verification: those programs that when run
test themselves. Our test suite includes these. test themselves. Our test suite includes these.
And Python comes with another a set of programs like this: its test And Python comes with another a set of programs like this: its test
suite for the standard library. We have some code in :code:`test/stdlib` to suite for the standard library. We have some code in :code:`test/stdlib` to
facilitate this kind of checking too. facilitate this kind of checking too.
Known Bugs/Restrictions Known Bugs/Restrictions
----------------------- -----------------------
The biggest known and possibly fixable (but hard) problem has to do The biggest known and possibly fixable (but hard) problem has to do
with handling control flow. (Python has probably the most diverse and with handling control flow. (Python has probably the most diverse and
screwy set of compound statements I've ever seen; there screwy set of compound statements I've ever seen; there
are "else" clauses on loops and try blocks that I suspect many are "else" clauses on loops and try blocks that I suspect many
programmers don't know about.) programmers don't know about.)
All of the Python decompilers that I have looked at have problems All of the Python decompilers that I have looked at have problems
decompiling Python's control flow. In some cases we can detect an decompiling Python's control flow. In some cases we can detect an
erroneous decompilation and report that. erroneous decompilation and report that.
Python support is pretty good for Python 2 Python support is pretty good for Python 2
On the lower end of Python versions, decompilation seems pretty good although On the lower end of Python versions, decompilation seems pretty good although
we don't have any automated testing in place for Python's distributed tests. we don't have any automated testing in place for Python's distributed tests.
Also, we don't have a Python interpreter for versions 1.6, and 2.0. Also, we don't have a Python interpreter for versions 1.6, and 2.0.
In the Python 3 series, Python support is strongest around 3.4 or In the Python 3 series, Python support is strongest around 3.4 or
3.3 and drops off as you move further away from those versions. Python 3.3 and drops off as you move further away from those versions. Python
3.0 is weird in that it in some ways resembles 2.6 more than it does 3.0 is weird in that it in some ways resembles 2.6 more than it does
@@ -218,43 +218,43 @@ Description: |buildstatus| |Pypi Installs| |Latest Version| |Supported Python V
:code:`EXTENDED_ARG` instructions, additional jump optimization has been :code:`EXTENDED_ARG` instructions, additional jump optimization has been
added. So in sum handling control flow by ad hoc means as is currently added. So in sum handling control flow by ad hoc means as is currently
done is worse. done is worse.
Between Python 3.5, 3.6, 3.7 there have been major changes to the Between Python 3.5, 3.6, 3.7 there have been major changes to the
:code:`MAKE_FUNCTION` and :code:`CALL_FUNCTION` instructions. :code:`MAKE_FUNCTION` and :code:`CALL_FUNCTION` instructions.
Python 3.8 removes :code:`SETUP_LOOP`, :code:`SETUP_EXCEPT`, Python 3.8 removes :code:`SETUP_LOOP`, :code:`SETUP_EXCEPT`,
:code:`BREAK_LOOP`, and :code:`CONTINUE_LOOP`, instructions which may :code:`BREAK_LOOP`, and :code:`CONTINUE_LOOP`, instructions which may
make control-flow detection harder, lacking the more sophisticated make control-flow detection harder, lacking the more sophisticated
control-flow analysis that is planned. We'll see. control-flow analysis that is planned. We'll see.
Currently not all Python magic numbers are supported. Specifically in Currently not all Python magic numbers are supported. Specifically in
some versions of Python, notably Python 3.6, the magic number has some versions of Python, notably Python 3.6, the magic number has
changes several times within a version. changes several times within a version.
**We support only released versions, not candidate versions.** Note **We support only released versions, not candidate versions.** Note
however that the magic of a released version is usually the same as however that the magic of a released version is usually the same as
the *last* candidate version prior to release. the *last* candidate version prior to release.
There are also customized Python interpreters, notably Dropbox, There are also customized Python interpreters, notably Dropbox,
which use their own magic and encrypt bytecode. With the exception of which use their own magic and encrypt bytecode. With the exception of
the Dropbox's old Python 2.5 interpreter this kind of thing is not the Dropbox's old Python 2.5 interpreter this kind of thing is not
handled. handled.
We also don't handle PJOrion_ or otherwise obfuscated code. For We also don't handle PJOrion_ or otherwise obfuscated code. For
PJOrion try: PJOrion Deobfuscator_ to unscramble the bytecode to get PJOrion try: PJOrion Deobfuscator_ to unscramble the bytecode to get
valid bytecode before trying this tool; pydecipher_ might help with that. valid bytecode before trying this tool; pydecipher_ might help with that.
This program can't decompile Microsoft Windows EXE files created by This program can't decompile Microsoft Windows EXE files created by
Py2EXE_, although we can probably decompile the code after you extract Py2EXE_, although we can probably decompile the code after you extract
the bytecode properly. `Pydeinstaller <https://github.com/charles-dyfis-net/pydeinstaller>`_ may help with unpacking Pyinstaller bundlers. the bytecode properly. `Pydeinstaller <https://github.com/charles-dyfis-net/pydeinstaller>`_ may help with unpacking Pyinstaller bundlers.
Handling pathologically long lists of expressions or statements is Handling pathologically long lists of expressions or statements is
slow. We don't handle Cython_ or MicroPython which don't use bytecode. slow. We don't handle Cython_ or MicroPython which don't use bytecode.
There are numerous bugs in decompilation. And that's true for every There are numerous bugs in decompilation. And that's true for every
other CPython decompiler I have encountered, even the ones that other CPython decompiler I have encountered, even the ones that
claimed to be "perfect" on some particular version like 2.4. claimed to be "perfect" on some particular version like 2.4.
As Python progresses decompilation also gets harder because the As Python progresses decompilation also gets harder because the
compilation is more sophisticated and the language itself is more compilation is more sophisticated and the language itself is more
sophisticated. I suspect that attempts there will be fewer ad-hoc sophisticated. I suspect that attempts there will be fewer ad-hoc
@@ -265,28 +265,28 @@ Description: |buildstatus| |Pypi Installs| |Latest Version| |Supported Python V
project is better funded, I do not intend to make any serious effort project is better funded, I do not intend to make any serious effort
to support Python versions 3.8 or 3.9, including bugs that might come to support Python versions 3.8 or 3.9, including bugs that might come
in. I imagine at some point I may be interested in it. in. I imagine at some point I may be interested in it.
You can easily find bugs by running the tests against the standard You can easily find bugs by running the tests against the standard
test suite that Python uses to check itself. At any given time, there are test suite that Python uses to check itself. At any given time, there are
dozens of known problems that are pretty well isolated and that could dozens of known problems that are pretty well isolated and that could
be solved if one were to put in the time to do so. The problem is that be solved if one were to put in the time to do so. The problem is that
there aren't that many people who have been working on bug fixing. there aren't that many people who have been working on bug fixing.
Some of the bugs in 3.7 and 3.8 are simply a matter of back-porting Some of the bugs in 3.7 and 3.8 are simply a matter of back-porting
the fixes in decompyle3. Volunteers are welcome to do so. the fixes in decompyle3. Volunteers are welcome to do so.
You may run across a bug, that you want to report. Please do so after You may run across a bug, that you want to report. Please do so after
reading `How to report a bug reading `How to report a bug
<https://github.com/rocky/python-uncompyle6/blob/master/HOW-TO-REPORT-A-BUG.md>`_ and <https://github.com/rocky/python-uncompyle6/blob/master/HOW-TO-REPORT-A-BUG.md>`_ and
follow the `instructions when opening an issue <https://github.com/rocky/python-uncompyle6/issues/new?assignees=&labels=&template=bug-report.md>`_. follow the `instructions when opening an issue <https://github.com/rocky/python-uncompyle6/issues/new?assignees=&labels=&template=bug-report.md>`_.
Be aware that it might not get my attention for a while. If you Be aware that it might not get my attention for a while. If you
sponsor or support the project in some way, I'll prioritize your sponsor or support the project in some way, I'll prioritize your
issues above the queue of other things I might be doing instead. issues above the queue of other things I might be doing instead.
See Also See Also
-------- --------
* https://github.com/rocky/python-decompile3 : Much smaller and more modern code, focusing on 3.7 and 3.8. Changes in that will get migrated back here. * https://github.com/rocky/python-decompile3 : Much smaller and more modern code, focusing on 3.7 and 3.8. Changes in that will get migrated back here.
* https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique than what is used here. Currently unmaintained. * https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique than what is used here. Currently unmaintained.
* https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Includes some fixes like supporting function annotations. Currently unmaintained. * https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Includes some fixes like supporting function annotations. Currently unmaintained.
@@ -297,8 +297,8 @@ Description: |buildstatus| |Pypi Installs| |Latest Version| |Supported Python V
* https://github.com/rocky/python-xasm : Cross Python version assembler * https://github.com/rocky/python-xasm : Cross Python version assembler
* https://github.com/rocky/python-uncompyle6/wiki : Wiki Documents which describe the code and aspects of it in more detail * https://github.com/rocky/python-uncompyle6/wiki : Wiki Documents which describe the code and aspects of it in more detail
* https://github.com/zrax/pycdc : The README for this C++ code says it aims to support all versions of Python. You can aim your slign shot for the moon too, but I doubt you are going to hit it. This code is best for Python versions around 2.7 and 3.3 when the code was initially developed. Accuracy for current versions of Python3 and early versions of Python is lacking. Without major effort, it is unlikely it can be made to support current Python 3. See its `issue tracker <https://github.com/zrax/pycdc/issues>`_ for details. Currently lightly maintained. * https://github.com/zrax/pycdc : The README for this C++ code says it aims to support all versions of Python. You can aim your slign shot for the moon too, but I doubt you are going to hit it. This code is best for Python versions around 2.7 and 3.3 when the code was initially developed. Accuracy for current versions of Python3 and early versions of Python is lacking. Without major effort, it is unlikely it can be made to support current Python 3. See its `issue tracker <https://github.com/zrax/pycdc/issues>`_ for details. Currently lightly maintained.
.. _Cython: https://en.wikipedia.org/wiki/Cython .. _Cython: https://en.wikipedia.org/wiki/Cython
.. _trepan: https://pypi.python.org/pypi/trepan3k .. _trepan: https://pypi.python.org/pypi/trepan3k
.. _compiler: https://github.com/rocky/python-uncompyle6/wiki/How-does-this-code-work%3F .. _compiler: https://github.com/rocky/python-uncompyle6/wiki/How-does-this-code-work%3F
@@ -323,9 +323,9 @@ Description: |buildstatus| |Pypi Installs| |Latest Version| |Supported Python V
.. |Latest Version| image:: https://badge.fury.io/py/uncompyle6.svg .. |Latest Version| image:: https://badge.fury.io/py/uncompyle6.svg
:target: https://badge.fury.io/py/uncompyle6 :target: https://badge.fury.io/py/uncompyle6
.. |Pypi Installs| image:: https://pepy.tech/badge/uncompyle6/month .. |Pypi Installs| image:: https://pepy.tech/badge/uncompyle6/month
Platform: UNKNOWN Platform: uncompyle6
Classifier: Development Status :: 5 - Production/Stable Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3) Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)

View File

@@ -1,64 +0,0 @@
[build-system]
requires = [
"setuptools>=61.2",
]
build-backend = "setuptools.build_meta"
[project]
authors = [
{name = "Rocky Bernstein", email = "rb@dustyfeet.com"},
]
name = "uncompyle6"
description = "Python cross-version byte-code library and disassembler"
dependencies = [
"click",
"spark-parser >= 1.8.9, < 1.9.0",
"xdis >= 6.0.8, < 6.2.0",
]
readme = "README.rst"
license = {text = "GPL"}
keywords = ["Python bytecode", "bytecode", "disassembler"]
classifiers = [
"Development Status :: 5 - Production/Stable",
"Intended Audience :: Developers",
"License :: OSI Approved :: MIT License",
"Programming Language :: Python",
"Topic :: Software Development :: Libraries :: Python Modules",
"Programming Language :: Python :: 2.4",
"Programming Language :: Python :: 2.5",
"Programming Language :: Python :: 2.6",
"Programming Language :: Python :: 2.7",
"Programming Language :: Python :: 3.0",
"Programming Language :: Python :: 3.1",
"Programming Language :: Python :: 3.2",
"Programming Language :: Python :: 3.3",
"Programming Language :: Python :: 3.4",
"Programming Language :: Python :: 3.5",
"Programming Language :: Python :: 3.6",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
]
dynamic = ["version"]
[project.urls]
Homepage = "https://github.com/rocky/python-uncompyle6"
Downloads = "https://github.com/rocky/python-uncompyle6/releases"
[project.optional-dependencies]
dev = [
"pre-commit",
"pytest",
]
[project.scripts]
uncompyle6 = "uncompyle6.bin.uncompile:main_bin"
uncompyle6-tokenize = "uncompyle6.bin.pydisassemble:main"
[tool.setuptools.dynamic]
version = {attr = "uncompyle6.version.__version__"}

View File

@@ -1,5 +1,8 @@
#!/usr/bin/env python #!/usr/bin/env python
"""Setup script for the 'uncompyle6' distribution.""" """Setup script for the 'uncompyle6' distribution."""
import sys
import setuptools
SYS_VERSION = sys.version_info[0:2] SYS_VERSION = sys.version_info[0:2]
if not ((3, 3) <= SYS_VERSION < (3, 6)): if not ((3, 3) <= SYS_VERSION < (3, 6)):
@@ -25,6 +28,38 @@ if not ((3, 3) <= SYS_VERSION < (3, 6)):
) )
print(mess) print(mess)
raise Exception(mess) raise Exception(mess)
from setuptools import setup
setup(packages=["uncompyle6"]) from __pkginfo__ import (
__version__,
author,
author_email,
classifiers,
entry_points,
install_requires,
license,
long_description,
modname,
py_modules,
short_desc,
web,
zip_safe,
)
setuptools.setup(
author=author,
author_email=author_email,
classifiers=classifiers,
description=short_desc,
entry_points=entry_points,
install_requires=install_requires,
license=license,
long_description=long_description,
long_description_content_type="text/x-rst",
name=modname,
packages=setuptools.find_packages(),
py_modules=py_modules,
test_suite="nose.collector",
url=web,
version=__version__,
zip_safe=zip_safe,
)