Compare commits

..

9 Commits

Author SHA1 Message Date
rocky
ab6d322eca Get ready for release 2.9.7 2016-12-04 14:09:53 -05:00
rocky
1a8a0df107 Merge branch 'master' into python-2.4 2016-12-04 13:40:06 -05:00
rocky
0a37709b0a CircleCI build 2016-11-24 05:41:31 -05:00
rocky
98cd1417df Remove dup Python 3 grammar rule 2016-11-24 05:36:43 -05:00
rocky
460069ceaa Bug in 2.4 "if" dectection and...
Wrong language used in old-style exceptions: use "except Error,e" not
"except Error(e)""
2016-11-24 05:15:35 -05:00
rocky
316aa44f23 Python 2.6 grammary bug and..
__pkginfo.py__: Bump spark_parser version for parse_flags 'dups'
2016-11-24 04:09:32 -05:00
rocky
7133540c23 Make work on 2.4 2016-11-23 08:26:12 -05:00
rocky
590231741d Merge branch 'come-from-type' into python-2.4 2016-11-23 07:54:18 -05:00
rocky
a9349b8f3d Making it run on Python 2.4 and 2.5 2016-11-23 07:53:51 -05:00
1118 changed files with 13356 additions and 43542 deletions

View File

@@ -1,28 +0,0 @@
# THis is an EditorConfig file
# https://EditorConfig.org
root = true
[*]
end_of_line = lf
insert_final_newline = true
charset = utf-8
indent_style = tab
indent_size = 4
insert_final_newline = true
[*.yml]
indent_style = space
indent_size = 2
end_of_line = lf
insert_final_newline = true
[*.py]
indent_style = space
indent_size = 4
end_of_line = lf
insert_final_newline = true
# Tab indentation (no size specified)
[Makefile]
indent_style = tab

12
.github/FUNDING.yml vendored
View File

@@ -1,12 +0,0 @@
# These are supported funding model platforms
github: [rocky]
patreon: # Replace with a single Patreon username
open_collective: # Replace with a single Open Collective username
ko_fi: # Replace with a single Ko-fi username
tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
liberapay: rocky
issuehunt: # Replace with a single IssueHunt username
otechie: # Replace with a single Otechie username
custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']

View File

@@ -1,140 +0,0 @@
---
name: Bug report
about: Tell us about uncompyle6 bugs
---
<!-- __Note:__ If you are using this program to do something illegal - don't.
The issue may be flagged to make it easier for those looking for illegal activity.
If you are reporting a bug in decompilation, it will probably not be acted upon
unless it is narrowed to a small example. You may have to do some work remove
extraneous code from the source example. Most bugs can be expressed in 30 lines of
code.
Issues are not for asking questions about a problem you
are trying to solve that involve the use of uncompyle6 along the way,
although I may be more tolerant of this if you sponsor the project.
Bugs are also not for general or novice kind help on how to install
this Python program and its dependencies in your environment, or in
the way you would like to have it set up, or how to interpret a Python
traceback e.g. that winds up saying Python X.Y.Z is not supported.
For these kinds of things, you will save yourself time by asking
instead on forums like StackOverflow that are geared to helping people
for such general or novice kinds questions and tasks. And unless you
are a sponsor of the project, if your question seems to be of this
category, the issue may just be closed.
Also, the unless you are a sponsor of the project, it may take a
while, maybe a week or so, before the bug report is noticed, let alone
acted upon.
To set expectations, some legitimate bugs can take years to fix, but
they eventually do get fixed.
Funding the project was added to partially address the problem that there are
lots of people seeking help and reporting bugs, but few people who are
willing or capable of providing help or fixing bugs.
Tasks or the kinds of things others can do, but you can't do or don't
want to do yourself are typically the kind of thing that you pay
someone to do, especially when you are the primary beneficiary of the
work, or the task is complex, long, or tedious. If your code is over
30 lines long, it fits into this category.
See also https://github.com/rocky/python-uncomp[yle6/blob/master/HOW-TO-REPORT-A-BUG.md ?
-->
<!--
Please remove any of the optional sections if they are not applicable.
Prerequisites/Caveats
* Make sure the bytecode you have can be disassembled with a
disassembler and produces valid results.
* Try to make the bytecode that exhibits a bug as small as possible.
* Don't put bytecode and corresponding source code on any service that
requires registration to download. Instead attach it as a zip file.
* When you open a bug report there is no privacy. If you need privacy, then
contact me by email and explain who you are and the need for privacy.
But be mindful that you may be asked to sponsor the project for the
personal and private help that you are requesting.
* If the legitimacy of the activity is deemed suspicious, I may flag it as suspicious,
making the issue even more easy to detect.
Bug reports that violate the above may be discarded.
-->
## Description
<!-- Please add a clear and concise description of the bug. Try to narrow the problem down to the smallest that exhibits the bug.-->
## How to Reproduce
<!-- Please show both the *input* you gave and the
output you got in describing how to reproduce the bug:
or give a complete console log with input and output
```console
$ uncompyle6 <command-line-options>
...
$
```
Attach a zip file to the Python bytecode or a
gist with the information. If you have the correct source code, you
can add that too.
-->
## Output Given
<!--
Please include not just the error message but all output leading to the message which includes echoing input and messages up to the error.
For a command-line environment include command invocation and all the output produced.
If this is too long, then try narrowing the problem to something short.
-->
## Expected behavior
<!-- Add a clear and concise description of what you expected to happen. -->
## Environment
<!-- _This section sometimes is optional but helpful to us._
Please modify for your setup
- Uncompyle6 version: output from `uncompyle6 --version` or `pip show uncompyle6`
- xdis version: output from `pydisasm --version` or or `pip show xdis`
- Python version for the version of Python the byte-compiled the file: `python -c "import sys; print(sys.version)"` where `python` is the correct CPython or PyPy binary.
- OS and Version: [e.g. Ubuntu bionic]
-->
## Workarounds
<!-- If there is a workaround for the problem, describe that here. -->
## Priority
<!-- If this is important for a particular public good state that here.
If this is blocking some important activity let us know what activity it blocks.
Otherwise, we'll assume this has the lowest priority in addressing.
-->
## Additional Context
<!-- _This section is optional._
Add any other context about the problem here or special environment setup.
-->

View File

@@ -1 +0,0 @@
blank_issues_enabled: False

View File

@@ -1,22 +0,0 @@
---
name: Feature Request
about: Tell us about a new feature that you would like to see in uncompyle6
---
## Description
<!-- Add a short description of the feature. This might
include same input and output. -->
## Background
<!-- Add any additional background for the
feature, for example: user scenarios, or the value of the feature. -->
## Tests
<!-- _This section is optional._
Add text with suggestions on how to test the feature,
if it is not obvious.
-->

12
.gitignore vendored
View File

@@ -1,28 +1,18 @@
*.pyc
*.pyo
*.pyc
*_dis
*~
.mypy_cache
/.cache
/.eggs
/.hypothesis
/.idea
/.mypy_cache
/.pytest_cache
/.python-version
/.tox
/.venv*
/README
/__pkginfo__.pyc
/dist
/how-to-make-a-release.txt
/nose-*.egg
/pycharm-venv
/tmp
/uncompyle6.egg-info
/unpyc
/venv
ChangeLog
__pycache__
build
nohup.out

View File

@@ -1,11 +0,0 @@
[settings]
multi_line_output = 3
include_trailing_comma = True
force_grid_wrap = 0
use_parentheses = True
line_length = 88
known_crunch = cr, zz9d, zz9lib, pycrunch, silhouette
sections = FUTURE,STDLIB,THIRDPARTY,FIRSTPARTY,CRUNCH,LOCALFOLDER
default_section = THIRDPARTY
combine_as_imports = 1
profile = black

View File

@@ -1,21 +1,12 @@
language: python
python:
# - '3.5'
# - '2.7'
# - '3.4'
- '3.6'
- '3.8'
sudo: false
matrix:
include:
- python: '3.7'
dist: xenial # required for Python >= 3.7 (travis-ci/travis-ci#9069)
python:
- '2.7' # this is a cheat here because travis doesn't do 2.4-2.6
install:
# Remove the next line when xdis 6.0.0 is released
# - pip install git://github.com/rocky/python-xdis.git#egg=xdis
- pip install -e .
- pip install -r requirements.txt
- pip install -r requirements-dev.txt
script:

674
COPYING
View File

@@ -1,674 +0,0 @@
GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU General Public License is a free, copyleft license for
software and other kinds of works.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
the GNU General Public License is intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users. We, the Free Software Foundation, use the
GNU General Public License for most of our software; it applies also to
any other work released this way by its authors. You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
To protect your rights, we need to prevent others from denying you
these rights or asking you to surrender the rights. Therefore, you have
certain responsibilities if you distribute copies of the software, or if
you modify it: responsibilities to respect the freedom of others.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must pass on to the recipients the same
freedoms that you received. You must make sure that they, too, receive
or can get the source code. And you must show them these terms so they
know their rights.
Developers that use the GNU GPL protect your rights with two steps:
(1) assert copyright on the software, and (2) offer you this License
giving you legal permission to copy, distribute and/or modify it.
For the developers' and authors' protection, the GPL clearly explains
that there is no warranty for this free software. For both users' and
authors' sake, the GPL requires that modified versions be marked as
changed, so that their problems will not be attributed erroneously to
authors of previous versions.
Some devices are designed to deny users access to install or run
modified versions of the software inside them, although the manufacturer
can do so. This is fundamentally incompatible with the aim of
protecting users' freedom to change the software. The systematic
pattern of such abuse occurs in the area of products for individuals to
use, which is precisely where it is most unacceptable. Therefore, we
have designed this version of the GPL to prohibit the practice for those
products. If such problems arise substantially in other domains, we
stand ready to extend this provision to those domains in future versions
of the GPL, as needed to protect the freedom of users.
Finally, every program is threatened constantly by software patents.
States should not allow patents to restrict development and use of
software on general-purpose computers, but in those that do, we wish to
avoid the special danger that patents applied to a free program could
make it effectively proprietary. To prevent this, the GPL assures that
patents cannot be used to render the program non-free.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based
on the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
The Corresponding Source for a work in source code form is that
same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Use with the GNU Affero General Public License.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU Affero General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the special requirements of the GNU Affero General Public License,
section 13, concerning interaction through a network will apply to the
combination as such.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If the program does terminal interaction, make it output a short
notice like this when it starts in an interactive mode:
<program> Copyright (C) <year> <name of author>
This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, your program's commands
might be different; for a GUI interface, you would use an "about box".
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU GPL, see
<http://www.gnu.org/licenses/>.
The GNU General Public License does not permit incorporating your program
into proprietary programs. If your program is a subroutine library, you
may consider it more useful to permit linking proprietary applications with
the library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License. But first, please read
<http://www.gnu.org/philosophy/why-not-lgpl.html>.

5149
ChangeLog Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -1,93 +1,157 @@
# Introduction
This project has history of over 17 years spanning back to Python 1.5
This project started around 1999 spanning back to Python 1.5
There have been a number of people who have worked on this. I am awed
by the amount of work, number of people who have contributed to this,
and the cleverness in the code.
In the interest of shortening what is written here, I am going to start where we left off where [decompyle 2.4's history](https://github.com/rocky/decompile-2.4/blob/master/HISTORY.md) ends.
The below is an annotated history from talking to participants
involved and my reading of the code and sources cited.
For the earlier history up to 2006 and the code up until Python 2.4, which I find interesting, look at that link.
In 1998, John Aycock first wrote a grammar parser in Python,
eventually called SPARK, that was usable inside a Python program. This
code was described in the
[7th International Python Conference](http://legacy.python.org/workshops/1998-11/proceedings/papers/aycock-little/aycock-little.html). That
paper doesn't talk about decompilation, nor did John have that in mind
at that time. It does mention that a full parser for Python (rather
than the simple languages in the paper) was being considered.
Sometime around 2014 was the dawn of ["uncompyle" and PyPI](https://pypi.python.org/pypi/uncompyle/1.1) &mdash; the era of
public version control. Dan Pascu's code although not public used [darcs](http://darcs.net/) for version control. I converted the darcs repository to git and put this at [decompyle-2.4](https://github.com/rocky/decompile-2.4).
[This](http://pages.cpsc.ucalgary.ca/~aycock/spark/content.html#contributors)
contains a of people acknowledged in developing SPARK. What's amazing
about this code is that it is reasonably fast and has survived up to
Python 3 with relatively little change. This work was done in
conjunction with his Ph.D Thesis. This was finished around 2001. In
working on his thesis, John realized SPARK could be used to deparse
Python bytecode. In the fall of 1999, he started writing the Python
program, "decompyle", to do this.
# uncompyle, unpyc
To help with control structure deparsing the instruction sequence was
augmented with pseudo instruction COME_FROM. This code introduced
another clever idea: using table-driven semantics routines, using
format specifiers.
In contrast to _decompyle_ that went up to Python 2.4, _uncompyle_, at least in its final versions, runs only on Python 2.7. However it accepts bytecode back to Python 2.5. Thomas Grainger is the package owner of this, although Hartmut is still listed as the author.
The last mention of a release of SPARK from John is around 2002. As
released, although the Early Algorithm parser was in good shape, this
code was woefully lacking as serious Python deparser.
The project exists not only on [github](https://github.com/gstarnberger/uncompyle) but also on
[bitbucket](https://bitbucket.org/gstarnberger/uncompyle) and later the defunct [google
code](https://code.google.com/archive/p/unpyc/) under the name _unpyc_. The git/svn history goes back to 2009. Somewhere in there the name was changed from "decompyle" to "unpyc" by Keknehv, and then to "uncompyle" by Guenther Starnberger.
In the fall of 2000, Hartmut Goebel
[took over maintaining the code](https://groups.google.com/forum/#!searchin/comp.lang.python/hartmut$20goebel/comp.lang.python/35s3mp4-nuY/UZALti6ujnQJ). The
first subsequent public release announcement that I can find is
["decompyle - A byte-code-decompiler version 2.2 beta 1"](https://mail.python.org/pipermail/python-announce-list/2002-February/001272.html).
The name Thomas Grainger isn't found in (m)any of the commits in the several years of active development. First Keknehv worked on this up to Python 2.5 or so while accepting Python bytecode back to 2.0 or so. Then "hamled" made a few commits earlier on, while Eike Siewertsen made a few commits later on. But mostly "wibiti", and Guenther Starnberger got the code to where uncompyle2 was around 2012.
From the CHANGES file found in
[the tarball for that release](http://old-releases.ubuntu.com/ubuntu/pool/universe/d/decompyle2.2/decompyle2.2_2.2beta1.orig.tar.gz),
it appears that Hartmut did most of the work to get this code to
accept the full Python language. He added precedence to the table
specifiers, support for multiple versions of Python, the
pretty-printing of docstrings, lists, and hashes. He also wrote test and verification routines of
deparsed bytecode, and used this in an extensive set of tests that he also wrote. He could verify against the entire Python library.
While John Aycock and Hartmut Goebel were well versed in compiler technology, those that have come afterwards don't seem to have been as facile in it. Furthermore, documentation or guidance on how the decompiler code worked, comparison to a conventional compiler pipeline, how to add new constructs, or debug grammars was weak. Some of the grammar tracing and error reporting was a bit weak as well.
decompyle2.2 was packaged for Debian (sarge) by
[Ben Burton around 2002](https://packages.qa.debian.org/d/decompyle.html). As
it worked on Python 2.2 only long after Python 2.3 and 2.4 were in
widespread use, it was removed.
Given this, perhaps it is not surprising that subsequent changes tended to shy away from using the built-in compiler technology mechanisms and addressed problems and extensions by some other means.
[Crazy Compilers](http://www.crazy-compilers.com/decompyle/) offers a
byte-code decompiler service for versions of Python up to 2.6. As
someone who worked in compilers, it is tough to make a living by
working on compilers. (For example, based on
[John Aycock's recent papers](http://pages.cpsc.ucalgary.ca/~aycock/)
it doesn't look like he's done anything compiler-wise since SPARK). So
I hope people will use the crazy-compilers service. I wish them the
success that his good work deserves.
Specifically, in `uncompyle`, decompilation of python bytecode 2.5 & 2.6 is done by transforming the byte code into a pseudo-2.7 Python bytecode and is based on code from Eloi Vanderbeken. A bit of this could have been easily added by modifying grammar rules.
Dan Pascu did a bit of work from late 2004 to early 2006 to get this
code to handle first Python 2.3 and then 2.4 bytecodes. Because of
jump optimization introduced in the CPython bytecode compiler at that
time, various JUMP instructions were classifed as going backwards, and
COME FROM instructions were reintroduced. See
RELEASE-2.4-CHANGELOG.txt for more details here. There wasn't a public
release of RELEASE-2.4 and bytecodes other than Python 2.4 weren't
supported. Dan says the Python 2.3 version could verify the entire
python library.
Next we get to ["uncompyle" and
PyPI](https://pypi.python.org/pypi/uncompyle/1.1) and the era of
public version control. (Dan's code although not public used
[darcs](http://darcs.net/) for version control.)
In contrast to _decompyle_, _uncompyle_ at least in its final versions,
runs only on Python 2.7. However it accepts bytecode back to Python
2.5. Thomas Grainger is the package owner of this, although Hartmut is
still listed as the author.
The project exists not only on
[github](https://github.com/gstarnberger/uncompyle) but also on
[bitbucket](https://bitbucket.org/gstarnberger/uncompyle) and later
the defunct [google
code](https://code.google.com/archive/p/unpyc/). The git/svn history
goes back to 2009. Somewhere in there the name was changed from
"decompyle" to "unpyc" by Keknehv, and then to "uncompyle" by Guenther Starnberger.
The name Thomas Grainger isn't found in (m)any of the commits in the
several years of active development. First Keknehv worked on this up
to Python 2.5 or so while acceping Python bytecode back to 2.0 or
so. Then hamled made a few commits earler on, while Eike Siewertsen
made a few commits later on. But mostly wibiti, and Guenther
Starnberger got the code to where uncompyle2 was around 2012.
In uncompyle2 decompilation of python bytecode 2.5 & 2.6 is done by
transforming the byte code into a a pseudo 2.7 python bytecode and is
based on code from Eloi Vanderbeken.
This project, uncompyle6, abandons that approach for various
reasons. However the main reason is that we need offsets in fragment
deparsing to be exactly the same, and the transformation process can
remove instructions. Adding instructions with psuedo_offsets is
however okay.
Uncompyle6, however owes its existence to the fork of uncompyle2 by
Myst herie (Mysterie) whose first commit picks up at
2012. I chose this since it seemed to have been at that time the most
actively, if briefly, worked on. Also starting around 2012 is Dark
Fenx's uncompyle3 which I used for inspiration for Python3 support.
I started working on this late 2015, mostly to add fragment support.
In that, I decided to make this runnable on Python 3.2+ and Python 2.6+
while, handling Python bytecodes from Python versions 2.5+ and
3.2+. In doing so, it has been expedient to separate this into three
projects: load loading and disassembly (xdis), parsing and tree
building (spark_parser), and grammar and semantic actions for
decompiling (uncompyle6).
# uncompyle2, uncompyle3, uncompyle6
Over the many years, code styles and Python features have
changed. However brilliant the code was and still is, it hasn't really
had a single public active maintainer. And there have been many forks
of the code. I have spent a great deal of time trying to organize and
modularize the code so that it can handle more Python versions more
gracefully (with still only moderate success).
`Uncompyle6`, which I started in 2015, owes its existence to the fork of [uncompyle2](https://github.com/Mysterie/uncompyle2) by Myst herie (Mysterie) whose first commit picks up at 2012. I chose this since it seemed to have been at that time the most actively, if briefly, worked on. Also starting around 2012 is Dark Fenx's [uncompyle3](https://github.com/DarkFenX/uncompyle3) which I used for inspiration for Python3 support.
I started working on this late 2015, mostly to add fragment support. In that, I decided to make this runnable on Python 3.2+ and Python 2.6+ while handling Python bytecodes from Python versions 2.5+ and
3.2+. In doing so, it was expedient to separate this into three projects:
* marshaling/unmarshaling, bytecode loading and disassembly ([xdis](https://pypi.python.org/pypi/xdis)),
* parsing and tree building ([spark_parser](https://pypi.python.org/pypi/spark_parser)),
* this project - grammar and semantic actions for decompiling
([uncompyle6](https://pypi.python.org/pypi/uncompyle6)).
`uncompyle6`, abandons the idea found in some 2.7 version of `uncompyle` that support Python 2.6 and 2.5 by trying to rewrite opcodes at the bytecode level.
Having a grammar per Python version is simpler to maintain, cleaner and it scales indefinitely.
Over the many years, code styles and Python features have changed. However brilliant the code was and still is, it hasn't really had a single public active maintainer. And there have been many forks of the code.
That this code has been in need of an overhaul has been recognized by the Hartmut more than two decades ago.
That it has been in need of an overhaul has been recognized by the
Hartmut a decade an a half ago:
[decompyle/uncompile__init__.py](https://github.com/gstarnberger/uncompyle/blob/master/uncompyle/__init__.py#L25-L26)
NB. This is not a masterpiece of software, but became more like a hack.
Probably a complete rewrite would be sensefull. hG/2000-12-27
In 2021, I created three git branches in order to allow the decompiler to run on a wide variety of Python versions from 2.4 up to 3.10. (Note this doesn't mean we decompile these versions. In fact we decompile starting from Python 1.0 up to Python 3.8 and no later than that.)
This project deparses using an Early-algorithm parse with lots of
massaging of tokens and the grammar in the scanner
phase. Early-algorithm parsers are context free and tend to be linear
if the grammar is LR or left recursive.
Using the separate git branches allows me to continually improve the coding style and add feature support while still supporting older Pythons. Supporting older Pythons is nice (but not strictly necessary) when you want to debug decompilation on older Pythons.
Another approach that doesn't use grammars is to do something like
simulate execution symbolically and build expression trees off of
stack results. The two important projects that work this way are
[unpyc3](https://code.google.com/p/unpyc3/) and most especially
[pycdc](https://github.com/zrax/pycdc) The latter project is largely
by Michael Hansen and Darryl Pogue. If they supported getting
source-code fragments and I could call it from Python, I'd probably
ditch this and use that. From what I've seen, the code runs blindingly
fast and spans all versions of Python.
I have spent a great deal of time trying to organize, modularize and even modernize the code so that it can handle more Python versions more gracefully (with still only moderate success).
Tests for the project have been, or are being, culled from all of the
projects mentioned.
Tests for the project have been, or are being, culled from all of the projects mentioned above or below. Quite a few have been added to improve grammar coverage and to address the numerous bugs that have been encountered.
# unpyc3 and pydc
Another approach to decompiling, and one that doesn't use grammars is to do something like simulate execution symbolically and build expression trees off of stack results. Control flow in that approach
still needs to be handled somewhat ad hoc. The two important projects that work this way are [unpyc3](https://code.google.com/p/unpyc3/) and most especially [pycdc](https://github.com/zrax/pycdc) The latter
project is largely by Michael Hansen and Darryl Pogue. If they supported getting source-code fragments, did a better job in supporting Python more fully, and had a way I could call it from Python, I'd probably would have ditched this and used that. The code runs blindingly fast and spans all versions of Python, although more recently Python 3 support has been lagging. The code is impressive for its smallness given that it covers many versions of Python. However, I think it has reached a scalability issue, same as all the other efforts. To handle Python versions more accurately, I think that code base will need to have a lot more code specially which specializes for Python versions. And then it will run into a modularity problem.
# So you want to write a decompiler for Python?
If you think, as I am sure will happen in the future, "hey, I can just write a decompiler from scratch and not have to deal with all of the complexity in uncompyle6", think again. What is likely to happen is that you'll get at best a 90% solution working for a single Python release that will be obsolete in about a year, and more obsolete each subsequent year.
Writing a decompiler for Python gets harder as it Python progresses. Writing decompiler for Python 3.7 isn't as easy as it was for Python 2.2. For one thing, now that Python has a well-established AST, that opens another interface by which code can be improved.
In Python 3.10 I am seeing (for the first time?) bytecode getting moved around so that it is no longer the case that line numbers have to be strictly increasing as bytecode offsets increase. And I am seeing dead code appear as well.
That said, if you still feel you want to write a single version decompiler, look at the test cases in this project and talk to me. I may have some ideas that I haven't made public yet. See also what I've written about the on how this code works and on [decompilation in dynamic runtime languages](http://rocky.github.io/Deparsing-Paper.pdf) in general.
# Earley Algorithm Parser
This project deparses using an Earley-algorithm parse. But in order to do this accurately, the process of tokenization is a bit more involved in the scanner. We don't just disassemble bytecode and use the opcode name. That aspect hasn't changed from the very first decompilers. However understanding _what_ information needs to be made explicit and what pseudo instructions to add that accomplish this has taken some time to understand.
Earley-algorithm parsers have gotten negative press, most notably by the dragon book. Having used this a bit, I am convinced having a system that handles ambiguous grammars is the right thing to do and matches the problem well. In practice the speed of the parser isn't a problem when one understand what's up. And this has taken a little while to understand.
Earley-algorithm parsers for context free languages or languages that are to a large extent context free and tend to be linear and the grammar steers towards left recursive rules. There is a technique for improving LL right recursion, but our parser doesn't have that yet.
The [decompiling paper](http://rocky.github.io/Deparsing-Paper.pdf) discusses these aspects in a more detail.
For a little bit of the history of changes to the Earley-algorithm parser, see the file [NEW-FEATURES.rst](https://github.com/rocky/python-spark/blob/master/NEW-FEATURES.rst) in the [python-spark github repository](https://github.com/rocky/python-spark).
NB. If you find mistakes, want corrections, or want your name added (or removed), please contact me.
NB. If you find mistakes, want corrections, or want your name added (or removed),
please contact me.

View File

@@ -1,238 +0,0 @@
<!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc -->
**Table of Contents**
- [Ethics](#ethics)
- [The importance of your bug report](#the-importance-of-your-bug-report)
- [The difficulty of the problem and your bug](#the-difficulty-of-the-problem-and-your-bug)
- [Is it really a bug?](#is-it-really-a-bug)
- [Do you have valid bytecode?](#do-you-have-valid-bytecode)
- [Semantic equivalence vs. exact source code](#semantic-equivalence-vs-exact-source-code)
- [What to send (minimum requirements)](#what-to-send-minimum-requirements)
- [What to send (additional helpful information)](#what-to-send-additional-helpful-information)
- [But I don't *have* the source code!](#but-i-dont-have-the-source-code)
- [But I don't *have* the source code and am incapable of figuring how to do a hand disassembly!](#but-i-dont-have-the-source-code-and-am-incapable-of-figuring-how-to-do-a-hand-disassembly)
- [Narrowing the problem](#narrowing-the-problem)
- [Karma](#karma)
- [Confidentiality of Bug Reports](#confidentiality-of-bug-reports)
<!-- markdown-toc end -->
TL;DR (too long; didn't read)
* Don't do something illegal. And don't ask me to do something illegal or help you do something illegal.
* We already have an infinite supply of decompilation bugs that need fixing, and an automated mechanism for finding more. Decompilation bugs get addressed by easiness to fix and by whim. If you expect yours to be fixed ahead of those, you need to justify why. You can ask for a hand-assisted decompilation, but that is expensive and beyond what most are willing to spend. A $100 fee is needed just to look at the bytecode.
* When asking for help, you may be asked for what you've tried on your own first. There are plenty of sources of information about this code.
* Bugs get fixed, slowly. Sometimes on the order of months or years. If you are looking for *timely* help or support, that is typically known as a _paid_ service.
* Submitting a bug or issue report that is likely to get acted upon may require a bit of effort on your part to make it easy for the problem solver. If you are not willing to do that, please don't waste your or our time. Bug report may be closed with about as much thought and care as apparent in the effort to create the bug. Supporting the project however, does increase the likelihood of your issue getting noticed and acted upon.
# Ethics
Do not use this program for unethical or illegal purposes. More detestable, at least to me, is asking for help to assist you in something that might not legitimate.
Don't use the issue tracker for such unethical or illegal solicitations. To try to stave off illegitimate behavior, you should note that the issue tracker, the code, and bugs mentioned in that are in the open: there is no
confidentiality. You may be asked about the authorship or claimed ownership of the bytecode. If I think something is not quite right, I may label the issue questionable which may make the it easier those who are looking for illegal activity.
# The importance of your bug report
For many open-source projects bugs where the expectation is that bugs are rare, reporting bugs in a *thoughtful* way can be helpful. See also [How to Ask Questions the Smart Way](http://www.catb.org/~esr/faqs/smart-questions.html).
In this project though, most of the bug reports boil down to the something like: I am trying to reverse engineer some code that I am not the author/owner and that person doesn't want me to have access to. I am hitting a problem somewhere along the line which might have to do with decompilation. But it could be something else like how the bytecode was extracted, some problem in deliberately obfuscated code, or the use some kind of Python bytecode version that isn't supported by the decompiler. Gee this stuff is complicated, here's an open source project, so maybe someone there will help me figure stuff out.
While you are free to report bugs, unless you sponsor the project, I may close them with about the same amount of effort spent that I think was used to open the report for them. And if you spent a considerable amount of time to create the bug report but didn't follow instructions given here and in the issue template, I am sorry in advance. Just go back, read, and follow instructions.
This project already has an infinite supply of bugs that have been narrowed to the most minimal form and where I have source code to compare against. And in the unlikely event this supply runs out, I have automated means for generating *another* infinite supply.
The task of justifying why addressing your bug is of use to the community, and why it should be prioritized over the others, is the bug reporter's responsibility.
While in the abstract, I have no problem answering questions about how to read a Python traceback or install Python software, or trying to understand what is going wrong in your particular setup, I am not a paid support person and there other things I'd rather be doing with my limited volunteer time. So save us both time, effort, and aggravation: use other avenues like StackOverflow. Again, justifying why you should receive unpaid help is the help requester's responsibility.
# The difficulty of the problem and your bug
This decompiler is a constant work in progress: Python keeps
changing, and so does its code generation.
There is no Python decompiler yet that I know about that will decompile everything. Overall, I think this one probably does the best job of *any* Python decompiler that handles such a wide range of versions.
But at any given time, there are a number of valid Python bytecode files that I know of that will cause problems. See, for example, the list in
[`test/stdlib/runtests.sh`](https://github.com/rocky/python-uncompyle6/blob/master/test/stdlib/runtests.sh).
There are far more bug reporters than there are bug fixers.
Unless you are a sponsor of this project, it may take a while, maybe a week or so, before the bug report is noticed, let alone acted upon. Things eventually get fixed, but it may take years. And if your bug hasn't been narrowed, it might happen as a result of some other bug fix.
# Is it really a bug?
## Do you have valid bytecode?
As mentioned in README.rst, this project doesn't handle obfuscated
code, release candidates, and the most recent versions of Python: version 3.9 and up. See README.rst for suggestions for how to remove some kinds of
obfuscation.
Checking if bytecode is valid is pretty simple: disassemble the code.
Python comes with a disassembly module called `dis`. A prerequisite
module for this package, `xdis` has a cross-python version
disassembler called `pydisasm`. Using that with the `-F extended` option, generally provides a more comprehensive disassembly than is provided by other disassemblers.
## Semantic equivalence vs. exact source code
Consider how Python compiles something like "(x*y) + 5". Early on Python creates an "abstract syntax tree" (AST) for this. And this is "abstract" in the sense that unimportant, redundant or unnecessary items have been removed. Here, this means that any notion that you wrote "x+y" in parenthesis is lost, since in this context they are unneeded. Also lost is the fact that the multiplication didn't have spaces around it while the addition did. It should not come as a surprise then that the bytecode which is derived from the AST also has no notion of such possible variation. Generally this kind of thing isn't noticed since the Python community has laid out a very rigid set of formatting guidelines; and it has largely beaten the community into compliance.
Almost all versions of Python can perform some sort of code
improvement that can't be undone. In earlier versions of Python it is
rare; in later Python versions, it is more common.
If the code emitted is semantically equivalent, then this isn't a bug.
For example the code might be
```python
if a:
if b:
x = 1
```
and we might produce:
```python
if a and b:
x = 1
```
These are equivalent. Sometimes
```
else:
if ...
```
may come out as `elif` or vice versa.
As mentioned in the README, It is possible that Python changes what
you write to be more efficient. For example, for:
```python
if True:
x = 5
```
Python will generate code like:
```python
x = 5
```
Even more extreme, if your code is:
```python
if False:
x = 1
y = 2
# ...
```
Python will eliminate the entire "if" statement.
So just because the text isn't the same, this does not necessarily mean there's a bug.
# What to send (minimum requirements)
The basic requirement is pretty simple:
* Python bytecode
* Python source text
Please don't put files on download services that one has to register
for or can't get to by issuing a simple `curl` or `wget`. If you can't
attach it to the issue, or create a github gist, then the code you are
sending is too large.
Also try to narrow the bug. See below.
# What to send (additional helpful information)
Some kind folks also give the invocation they used and the output
which usually includes an error message produced. This is
helpful. From this, I can figure out what OS you are running this on
and what version of *uncompyle6* was used. Therefore, if you _don't_
provide the input command and the output from that, please give:
* _uncompyle6_ version used
* OS that you used this on
* Python interpreter version used
## But I don't *have* the source code!
There is Python assembly code on parse errors, so simply by hand decompile that. To get a full disassembly, use `pydisasm` from the [xdis](https://pypi.python.org/pypi/xdis) package. Opcodes are described in the documentation for the [dis](https://docs.python.org/3.6/library/dis.html) module.
### But I don't *have* the source code and am incapable of figuring how to do a hand disassembly!
Well, you could learn. No one is born into this world knowing how to disassemble Python bytecode. And as Richard Feynman once said, "What one fool can learn, so can another."
If this is too difficult, or too time consuming, or not of interest to you, then you might consider [sponsoring](https://github.com/sponsors/rocky) the project. [Crazy
Compilers](http://www.crazy-compilers.com/decompyle/) offers a byte-code decompiler service for versions of Python up to 2.6. (If there are others around let me know and I'll list them here.) Don't be surprised if I ask you to pay for work (if I think the work is ethical) when you want me to work on your problem that I think isn't of interest or benefit to anyone but yourself or a small limited number of people, or I think the need is questionable.
# Narrowing the problem
I don't need or want the entire source code base for the file(s) or module(s) can't be decompiled. I just need those file(s) or module(s). If there are problems in several files, file a bug report for each file.
Python modules can get quite large, and usually decompilation problems
occur in a single function or maybe the main-line code but not any of
the functions or classes. So please chop down the source code by
removing those parts that do to decompile properly.
By doing this, you'll probably have a better sense of what exactly is
the problem. Perhaps you can find the boundary of what decompiles, and
what doesn't. That is useful. Or maybe the same file will decompile
properly on a neighboring version of Python. That is helpful too.
In sum, the more you can isolate or narrow the problem, the more
likely the problem will be fixed and fixed sooner.
# Karma
I realize that following the instructions given herein puts a bit of
burden on the bug reporter. This is justified since it attempts to balance
the burden and effort needed to fix the bug with the amount of effort to report the problem. And it attempts
to balance number of would-be bug reporters with the number of bug
fixers. Better bug reporters are more likely to move in the category
of bug fixers.
The barrier to reporting a big is pretty small: all you really need is
a github account, and the ability to type something after clicking
some buttons. So the reality is that many people just don't bother to
read these instructions, let alone follow it to any simulacrum.
That said, bugs sometimes get fixed even though these instructions are not followed.
I may take into consideration is the bug reporter's karma.
* Have you demonstrably contributed to open source? I may look at your github profile to see what contributions you have made, how popular those contributions are, or how popular you are.
* How appreciative are you? Have you starred this project that you are seeking help from? Have you starred _any_ github project? And the above two kind of feed into ...
* Attitude. Some people feel that they are doing me and the world a
great favor by just pointing out that there is a problem whose
solution would greatly benefit them. (This might account partially
for the fact that those that have this attitude often don't read or
follow instructions such as those given here.)
# Confidentiality of Bug Reports
When you report a bug, you are giving up confidentiality to the source
code and the byte code. However, I would imagine that if you have
narrowed the problem sufficiently, confidentiality of the little that
remains would not be an issue.
However feel free to remove any comments, and modify variable names
or constants in the source code.
If there is some legitimate reason to keep confidentiality, you can contact me by email to explain the extenuating circumstances. However I tend to discard without reading anonymous email.
Private consulting available via https://calendly.com/rb3216 rates: $150 for 30 minutes; $250 for 60 minutes.

22
LICENSE Normal file
View File

@@ -0,0 +1,22 @@
Copyright (c) 2015 by Rocky Bernstein
Copyright (c) 2000 by hartmut Goebel <h.goebel@crazy-compilers.com>
Copyright (c) 1998-2002 John Aycock
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

View File

@@ -1,8 +1,7 @@
include README.rst
include ChangeLog
include HISTORY.md
include HOW-TO-REPORT-A-BUG.md
include COPYING
include LICENSE
include Makefile
include requirements.txt
include requirements-dev.txt

View File

@@ -11,10 +11,7 @@ RM ?= rm
LINT = flake8
#EXTRA_DIST=ipython/ipy_trepan.py trepan
PHONY=all check check-2.7 check-3.4 \
clean distcheck pytest check-long check-short \
dist distclean lint flake8 test rmChangeLog clean_pyc \
2.6 5.0 5.3 5.6 5.8 7.2 7.3 check-short
PHONY=all check clean pytest check-long dist distclean lint flake8 test rmChangeLog clean_pyc
TEST_TYPES=check-long check-short check-2.7 check-3.4
@@ -30,36 +27,25 @@ check:
check-short: pytest
$(MAKE) -C test check-short
# Note for 2.6 use <=3.0.1 see requirements-dev.txt
#: Tests for Python 2.7, 3.3 and 3.4
check-2.6 check-2.7 check-3.3 check-3.4 check-3.5: pytest
check-2.7 check-3.3 check-3.4: pytest
$(MAKE) -C test $@
#: Tests for Python 3.2 and 3.5 - pytest doesn't work here
# Or rather 3.5 doesn't work not on Travis
check-3.0 check-3.1 check-3.2 check-3.6:
check-3.0 check-3.1 check-3.2 check-3.5 check-3.6:
$(MAKE) -C test $@
check-3.7: pytest
$(MAKE) -C test check
#:Tests for Python 2.6 (doesn't have pytest)
check-2.4 check-2.5 check-2.6:
$(MAKE) -C test $@
check-3.8:
$(MAKE) -C test check
#:PyPy 2.6.1 PyPy 5.0.1, or PyPy 5.8.0-beta0
#:PyPy 2.6.1 or PyPy 5.0.1
# Skip for now
2.6 5.0 5.3 5.6 5.8:
2.6 5.0 5.3:
#:PyPy pypy3-2.4.0 Python 3.6.1:
7.1 pypy-3.2 2.4:
$(MAKE) -C test $@
#:PyPy versions
7.2 7.3:
$(MAKE) -C test $@
#:pyston versions
2.3:
#:PyPy pypy3-2.4.0 Python 3:
pypy-3.2 2.4:
$(MAKE) -C test $@
#: Run py.test tests
@@ -72,13 +58,9 @@ clean: clean_pyc
(cd test && $(MAKE) clean)
#: Create source (tarball) and wheel distribution
dist: distcheck
dist:
$(PYTHON) ./setup.py sdist bdist_wheel
# perform some checks on the package via setup.py
distcheck:
$(PYTHON) ./setup.py check
#: Remove .pyc files
clean_pyc:
( cd uncompyle6 && $(RM) -f *.pyc */*.pyc )
@@ -105,7 +87,7 @@ bdist_egg:
#: Create binary wheel distribution
wheel:
bdist_wheel:
$(PYTHON) ./setup.py bdist_wheel

298
NEWS Normal file
View File

@@ -0,0 +1,298 @@
uncompyle6 2.9.6 2016-12-04
- Shorten Python3 grammars with + and *
this requires spark parser 1.5.1
- Add some AST reduction checks to improve
decompile accuracy. This too requires
spark parser 1.5.1
uncompyle6 2.9.6 2016-11-20
- Correct MANIFEST.in
- More AST grammar checking
- --linemapping option or linenumbers.line_number_mapping()
Shows correspondence of lines between source
and decompiled source
- Some control flow adjustments in code for 2.x.
This is probably an improvement in 2.6 and before.
For 2.7 things are just shuffled around a little. Sigh.
Overall I think we are getting more precise in
or analysis even if it is not always reflected
in the results.
- better control flow debugging output
- Python 2 and 3 detect structure code is more similar
- Handle Docstrings with embedded tiple quotes (""")
uncompyle6 2.9.5 2016-11-13
- Fix Python 3 bugs:
* improprer while 1 else
* docstring indent
* 3.3 default values in lambda expressions
* start 3.0 decompilation (needs newer xdis)
- Start grammar misparse checking
uncompyle6 2.9.4 2016-11-02
- Handle Python 3.x function annotations
- track def keywoard-parameter line-splitting in source code better
- bump min xdis version to mask previous xdis bug
uncompyle6 2.9.3 2016-10-26
Release forced by incompatiblity change in xdis 3.2.0.
- Python 3.1 bugs:
* handle "with ... as"
* handle "with"
* Start handling def (...) -> yy (has bugs still)
- DRY Python 3.x via inheritance
- Python 3.6 work (from Daniel Bradburn)
* Handle 3.6 buildstring
* Handle 3.6 handle single and multiple fstring better
uncompyle6 2.9.2 2016-10-15
- use source-code line breaks to assist in where to break
in tuples and maps
- Fix Python 1.5 decompyle bugs
- Fix some Python 2.6 and below bugs
- DRY fragments.py code a little
uncompyle6 2.9.1 2016-10-09
- Improved Python 1.5 decompiling
- Handle old-style pre Python 2.2 classes
uncompyle6 2.9.0 2016-10-09
- Use xdis 3.0.0 protocol load_module.
this Forces change in requirements.txt and _pkg_info_.py
- Start Python 1.5 decompiling; another round of work is needed to
remove bugs
- Simpify python 2.1 grammar
- Fix bug with -t ... Wasn't showing source text when -t option was given
- Fix 2.1-2.6 bug in list comprehension
uncompyle6 2.8.4 2016-10-08
- Python 3 disassembly bug fixes
- Python 3.6 fstring bug fixes (from moagstar)
- Python 2.1 disassembly
- COME_FROM suffixes added in Python3
- use .py extension in verification disassembly
uncompyle6 2.8.3 2016-09-11 live from NYC!
NOTE: this is possibly the last release before a major reworking of
control-flow structure detection is done.
- Lots of bug fixes in decompilation:
* 3.0 .. 3.4 whileTrue bug
* 3.x function declaration deparsing:
. 3.0 .. 3.2 *args processing
. 3.0 .. 3.2 call name and kwargs bug
. 3.0 .. getting parameter of *
. 3.0 .. handling varible number of args
. 3.0 .. "if" structure bugs
* 3.5+ if/else bugs
* 2.2-2.6 bugs
. try/except control flow
. a == b == c -like detection
. generator detection
. "while .. and" statement bugs
. handle "except <cond>, <var>"
. use older raise format in 2.x
- scanner "disassemble" is now "ingest". True disassembly is done by xdis
- Start accepting Python 3.1 bytecode
- Add --weak-verify option on test_pyenvlib and test_pythonlib. This
catches more bugs more easily
- bump xdis requirement so we can deparse dropbox 2.5 code
- Added H. Goebel's changes before 2.4 in DECOMPYLE-2.4-CHANGELOG.txt
uncompyle6 2.8.2 2016-08-29
- Handle Python 3.6 format string conversions !r, !s, !a
- Start to handle 3.1 bytecode
- Fix some PyPy translation bugs
- We now only handle 3.6.0a3+ since that is incompatible with 3.6 before that
uncompyle6 2.8.1 2016-08-20
- Add Python 2.2 decompilation
- Fix bugs
* PyPy LOOKUP_METHOD bug
* Python 3.6 FORMAT_VALUE handles expressions now
uncompyle6 2.8.0 2016-08-03
- Start Python 3.6 support (moagstar)
more work on PEP 498 needed
- tidy bytecode/word output
- numerous decompiling bugs fixed
- grammar testing started
- show magic number in deparsed output
- better grammar and semantic action segregation based
on python bytecode version
uncompyle6 2.7.1 2016-07-26
- PyPy bytecodes for 2.7 and 3.2 added
- Instruction formatting improved slightly
- 2.7 bytecode "continue" bug fixed
uncompyle6 2.7.0 2016-07-15
- Many Syntax and verifification bugs removed
tested on standard libraries from 2.3.7 to 3.5.1
and they all decompile and verify fine.
I'm sure there are more bugs though.
uncompyle6 2.6.2 2016-07-11 Manhattenhenge
- Extend bytecodes back to 2.3
- Fix bugs:
* 3.x and 2.7 set comprehensions,
* while1 loops
* continue statements
- DRY and segregate grammar more
uncompyle6 2.6.1 2016-07-08
- Go over Python 2.5 bytecode deparsing
all library programs now deparse
- Fix a couple bugs in 2.6 deparsing
uncompyle6 2.6.0 2016-07-07
- Improve Python 2.6 bytecode deparsing:
stdlib now will deparse something
- Better <2.6 vs. 2.7 grammar separation
- Fix some 2.7 deparsing bugs
- Fix bug in installing uncompyle6 script
- Doc improvments
uncompyle6 2.5.0 2016-06-22 Summer Solstace
- Much better Python 3.2-3.5 coverage.
3.4.6 is probably the best;3.2 and 3.5 are weaker
- Better AST printing with -t
- Better error reporting
- Better fragment offset tracking
- Some (much-needed) code refactoring
uncompyle6 2.4.0 2016-05-18 (in memory of Lewis Bernstein)
- Many Python 3 bugs fixed:
* Python 3.2 to 3.5 libaries largely
uncompyle and most verify
- pydisassembler:
* disassembles all code objects in a file
* can select showing bytecode before
or after uncompyle mangling, option -U
- DRY scanner code (but more is desired)
- Some code cleanup (but more is desired)
- Misc Bugs fixed:
* handle complex number unmarshaling
* Running on Python 2 to works on Python 3.5 bytecodes now
uncompyle6 2.3.5 and 2.3.6 2016-05-14
- Python 2 class decorator fix (thanks to Tey)
- Fix fragment parsing bugs
- Fix some Python 3 parsing bugs:
* Handling single in * parameter
* "while True"
* escape from for inside if
* yield expressions
- Correct history based on info from Dan Pascu
- Fix up pip packaging, ugh.
uncompyle6 2.3.4 2016-05-5
- More Python 3.5 parsing bugs addressed
- decompiling Python 3.5 from other Python versions works
- test from Python 3.2
- remove "__module__ = __name__" in 3.0 <= Python 3.2
uncompyle6 2.3.3 2016-05-3
- Fix bug in running uncompyle6 script on Python 3
- Speed up performance on deparsing long lists by grouping in chunks of 32 and 256 items
- DRY Python expressions between Python 2 and 3
uncompyle6 2.3.2 2016-05-1
- Add --version option standalone scripts
- Correct License information in package
- expose fns uncompyle_file, load_file, and load_module
- Start to DRY Python2 and Python3 grammars Separate out 3.2, and 3.5+
specific grammar code
- Fix bug in 3.5+ constant map parsing
uncompyle6 2.3.0, 2.3.1 2016-04-30
- Require spark_parser >= 1.1.0
uncompyle6 2.2.0 2016-04-30
- Spark is no longer here but pulled separate package spark_parse
- Python 3 parsing fixes
- More tests
uncompyle6 2.2.0 2016-04-02
- Support single-mode (in addtion to exec-mode) compilation
- Start to DRY Python 2 and Python 3 grammars
- Fix bug in if else ternary construct
- Fix bug in uncomplye6 -d and -r options (via lelicopter)
uncompyle6 2.1.3 2016-01-02
- Limited support for decompiling Python 3.5
- Improve Python 3 class deparsing
- Handle MAKE_CLOSURE opcode
- Start to DRY opcode code.
- increase test coverage
- fix misc small bugs and some improvements
uncompyle6 2.1.2 2015-12-31
- Fix cross-version Marshal loading
- Handle Python 3.3 . dotted class names
- Limited 3.5 support: allows deparsing other versions
- Refactor code more, misc bug fixes
uncompyle6 2.1.1 2015-12-27
- packaging issues
uncompyle6 2.1.0 2015-12-27
- Python 3.x deparsing much more solid
- Better cross-version deparsing
Some bugs squashed while other run rampant. Some code cleanup while
much more is yet needed. More tests added, but many more are needed.
uncompyle6 2.0.0 2015-12-11
Changes from uncompyle2
- Can give code fragments given an instruction offset. See
https://github.com/rocky/python-uncompyle6/wiki/Deparsing-technology-and-its-use-in-exact-location-reporting
- Runs under Python3. Decompiles Python 2.5-2.7 and some Python 3.2-3.4
- Allows for multiple Python grammars, specifically Python2 vs Python 3
- Add a cross-version Python disassembler command-line utility
- Add some py.test and start reorganizing tests
SPARK:
add option to show grammar rules applied
allow Python-style # comments in grammar
Runs on Python 3 and Python 2

1189
NEWS.md

File diff suppressed because it is too large Load Diff

View File

@@ -1,221 +0,0 @@
Introduction
============
The original versions of this code up until the time I started were
pretty awesome. You can get a sense of this by running it. For the
most part it was remarkably fast, and a single module with few dependencies.
Here I will largely give what are the major improvements over old code.
This also serves to outline a little bit about what is in this code.
See also `How does this code work? <https://github.com/rocky/python-uncompyle6/wiki/How-does-this-code-work%3F>`_.
Old Cool Features
==================
Before getting to the new stuff, I'll describe cool things that was there before.
I particularly liked the ability to show the assembly, grammar
reduction rules as they occurred, and the resulting parse tree. It is
neat that you could follow the process and steps that deparser takes,
and in this not only see the result how the bytecode corresponds to
the resulting source. Compare this with other Python decompilers.
And of course also neat was that this used a grammar and table-driven
approach to decompile.
Expanding decompilation to multiple Python Versions
==================================================
Aside from ``pycdc``, most of the Python decompilers handle a small
number of Python versions, if they supported more than one. And even
when more than one version is supported if you have to be running the
Python version that the bytecode was compiled for.
There main reason that you have to be running the Python bytecode
interpreter as the one you want to decompile largely stems from the
fact that Python's ``dis`` module is often what is used and that has this limitation.
``pycdc`` doesn't suffer this problem because it is written in C++,
not Python. Hartmut Goebel's code had provisions for multiple Python
versions running from an interpreter different from the one that was
running the decompiler. That however used compiled code in the process
was tied a bit to the Python C headers for a particular version.
You need to not only to account for different "marshal" and "unmarshal"
routines for the different Python versions, but also, as the Python versions
extend, you need a different code type as well.
Enter ``xdis``
--------------
To handle all of these problems, I split off the marshal loading
portion and disassembly routines into a separate module,
`xdis <https://pypi.org/project/xdis/>`_. This also allows older Pythons to have access to features
found in newer Pythons, such as parsing the bytecode, a uniform stream
of bytes, into a list of structured bytecode instructions.
Python 2.7's ``dis`` module doesn't has provide a instruction abstraction.
Therefore in ``uncompyle2`` and other earlier decompilers you see code with magic numbers like 4 in::
if end > jump_back+4 and code[end] in (JF, JA):
if code[jump_back+4] in (JA, JF):
if self.get_target(jump_back+4) == self.get_target(end):
self.fixed_jumps[pos] = jump_back+4
end = jump_back+4
elif target < pos:
self.fixed_jumps[pos] = jump_back+4
end = jump_back+4
and in other code -1 and 3 in::
if self.get_target(jmp) != start_else:
end_else = self.get_target(jmp)
if self.code[jmp] == JF:
self.fixed_jumps[jmp] = -1
self.structs.append({'type': 'except',
'start': i,
'end': jmp})
i = jmp + 3
All of that offset arithmetic is trying to find the next instruction
offset or the previous offset. Using a list of instructions you simply
take the ``offset`` field of the previous or next instruction.
The above code appears in the ``uncompyle2`` "Scanner" class in
service of trying to figure out control flow. Note also that there
isn't a single comment in there about what specifically it is trying
to do, the logic or that would lead one to be confident that this is
correct, let alone assumptions that are needed for this to be true.
While this might largely work for Python 2.7, and ``uncompyle2`` does
get control flow wrong sometimes, it is impossible to adapt code for
other versions of Python.
In addition adding an instruction structure, ``xdis`` adds various
flags and features that assist in working with instructions. In the
example above this replaces code like ``... in (JF, JA)`` which is
some sort of unconditional jump instruction.
Although not needed in the decompiler, ``xdis`` also has nicer
instruction print format. It can show you the bytes as well as the
interpreted instructions. It will interpret flag bits and packed
structures in operands so you don't have to. It can even do a limited
form of inspection at previous instructions to give a more complete
description of an operand. For example on ``LOAD_ATTR`` which loads
the attribute of a variable, often the variable name can be found as
the previous instruction. When that is the case the disassembler can
include that in the disassembly display for the ``LOAD_ATTR`` operand.
Python Grammar Isolation
------------------------
If you want to support multiple versions of Python in a manageable way
you really need to provide different grammars for the different
versions, in a grammar-based system. None of the published versions of
this decompiler did this.
If you look at the changes in this code, right now there are no
grammar changes needed between 1.0 to 1.3. (Some of this may be wrong
though since we haven't extensively tested these earliest Python versions
For Python 1.4 which is based off of the grammar for 1.5 though there
are number of changes, about 6 grammar rules. Later versions of though
we start to see larger upheaval and at certain places, especially
those where new opcodes are introduced, especially those that change
the way calls or exceptions get handled, we have major upheaval in the
grammar. It is not just that some rules get added, but we also need to
*remove* some grammar rules as well.
I have been largely managing this as incremental differences between versions.
However in the future I am leaning more towards totally separate grammars.
A well constructed grammar doesn't need to be that large.
When starting out a new version, we can just copy the grammar from the
prior version. Within a Python version though, I am breaking these
into composable pieces. In particular the grammar for handling what
can appear as the body of a lambda, is a subset of the full Python
language. The language allowed in an ``eval`` is also a subset of the
full Python language, as are what can appear in the various
compilation modes like "single" versus "exec".
Another nice natural self-contain grammar section is what can appear
in list comprehensions and generators. The bodies of these are
generally represented in a self-contained code block.
Often in decompilation you may be interested not just in decompiling
the entire code but you may be interested in only focusing on a
specific part of the code. And if there is a problem in decompiling
the entire piece of code, having these smaller breaking points can be
of assistance.
Other Modularity
----------------
Above we have mentioned the need for separate grammars or to isolate
these per versions. But there are other major pieces that make up this
decompiler. In particular there is a scanner and the source code
generation part.
Even though differences in version that occur in disassembly are
handled by ``xdis``, we still have to do conversion of that to a token
stream for parsing. So the scanners are again broken out per version
with various OO mechanisms for reusing code. The same is true for
source code generation.
Expanding decompiler availability to multiple Python Versions
--------------------------------------------------------------
Above we mention decompiling multiple versions of bytecode from a
single Python interpreter. We talk about having the decompiler
runnable from multiple versions of Python, independent of the set of
bytecode that the decompiler supports.
There are slight advantages in having a decompiler that runs the same
version as the code you are decompiling. The most obvious one is that
it makes it easy to test to see whether the decompilation correct
because you can run the decompiled code. Python comes with a suite of
Python programs that check themselves and that aspects of Python are
implemented correctly. These also make excellent programs to check
whether a program has decompiled correctly.
Aside from this, debugging can be easier as well. To assist
understanding bytecode and single stepping it see `x-python
<https://pypi.org/project/x-python/>`_ and the debugger for it
`trepan-xpy <https://pypi.org/project/trepanxpy/>`_.
Handling Language Drift
-----------------------
Given the desirability of having this code running on logs of Python
versions, how can we get this done?
The solution used here is to have several git branches of the
code. Right now there are 3 branches. Each branch handles works across
3 or so different releases of Python. In particular one branch handles
Python 2.4 to 2.7 Another handles Python 3.3 to 3.5, and the master
branch handles 3.6 to 3.10. (Again note that the 3.9 and 3.10
decompilers do not decompile Python 3.9 or 3.10, but they do handle
bytecode for all earlier versions.)
Cool features of the Parser
===========================
* reduction rule checking
* numbering tokens
* showing a stack of completions
Cool features Semantic Analysis
===============================
* ``--tree++`` (``-T``) option
* showing precedence
* See `Adding a tree transformation phase to uncompyle6 <https://github.com/rocky/python-uncompyle6/wiki/Adding-a-tree-transformation-phase-to-uncompyle6>`_
* following AST
* Fragment deparsing

361
PKG-INFO
View File

@@ -1,355 +1,10 @@
Metadata-Version: 1.1
Metadata-Version: 2.0
Name: uncompyle6
Version: 3.9.1
Summary: Python cross-version byte-code decompiler
Home-page: https://github.com/rocky/python-uncompyle6/
Author: Rocky Bernstein, Hartmut Goebel, John Aycock, and others
Version: 2.0.1
Summary: Python byte-code to source-code converter
Home-page: http://github.com/rocky/python-uncompyle6
Author: Rocky
Author-email: rb@dustyfeet.com
License: GPL3
Description: |buildstatus| |Pypi Installs| |Latest Version| |Supported Python Versions|
|packagestatus|
.. contents::
uncompyle6
==========
A native Python cross-version decompiler and fragment decompiler.
The successor to decompyle, uncompyle, and uncompyle2.
Introduction
------------
*uncompyle6* translates Python bytecode back into equivalent Python
source code. It accepts bytecodes from Python version 1.0 to version
3.8, spanning over 24 years of Python releases. We include Dropbox's
Python 2.5 bytecode and some PyPy bytecodes.
Why this?
---------
Ok, I'll say it: this software is amazing. It is more than your
normal hacky decompiler. Using compiler_ technology, the program
creates a parse tree of the program from the instructions; nodes at
the upper levels that look a little like what might come from a Python
AST. So we can really classify and understand what's going on in
sections of Python bytecode.
Building on this, another thing that makes this different from other
CPython bytecode decompilers is the ability to deparse just
*fragments* of source code and give source-code information around a
given bytecode offset.
I use the tree fragments to deparse fragments of code *at run time*
inside my trepan_ debuggers_. For that, bytecode offsets are recorded
and associated with fragments of the source code. This purpose,
although compatible with the original intention, is yet a little bit
different. See this_ for more information.
Python fragment deparsing given an instruction offset is useful in
showing stack traces and can be incorporated into any program that
wants to show a location in more detail than just a line number at
runtime. This code can be also used when source-code information does
not exist and there is just bytecode. Again, my debuggers make use of
this.
There were (and still are) a number of decompyle, uncompyle,
uncompyle2, uncompyle3 forks around. Many of them come basically from
the same code base, and (almost?) all of them are no longer actively
maintained. One was really good at decompiling Python 1.5-2.3, another
really good at Python 2.7, but that only. Another handles Python 3.2
only; another patched that and handled only 3.3. You get the
idea. This code pulls all of these forks together and *moves
forward*. There is some serious refactoring and cleanup in this code
base over those old forks. Even more experimental refactoring is going
on in decompyle3_.
This demonstrably does the best in decompiling Python across all
Python versions. And even when there is another project that only
provides decompilation for subset of Python versions, we generally do
demonstrably better for those as well.
How can we tell? By taking Python bytecode that comes distributed with
that version of Python and decompiling these. Among those that
successfully decompile, we can then make sure the resulting programs
are syntactically correct by running the Python interpreter for that
bytecode version. Finally, in cases where the program has a test for
itself, we can run the check on the decompiled code.
We use an automated processes to find bugs. In the issue trackers for
other decompilers, you will find a number of bugs we've found along
the way. Very few to none of them are fixed in the other decompilers.
Requirements
------------
The code in the git repository can be run from Python 2.4 to the
latest Python version, with the exception of Python 3.0 through
3.2. Volunteers are welcome to address these deficiencies if there a
desire to do so.
The way it does this though is by segregating consecutive Python versions into
git branches:
master
Python 3.6 and up (uses type annotations)
python-3.3-to-3.5
Python 3.3 through 3.5 (Generic Python 3)
python-2.4
Python 2.4 through 2.7 (Generic Python 2)
PyPy 3-2.4 and later works as well.
The bytecode files it can read have been tested on Python
bytecodes from versions 1.4, 2.1-2.7, and 3.0-3.8 and later PyPy
versions.
Installation
------------
You can install from PyPI using the name ``uncompyle6``::
pip install uncompyle6
To install from source code, this project uses setup.py, so it follows the standard Python routine::
$ pip install -e . # set up to run from source tree
or::
$ python setup.py install # may need sudo
A GNU Makefile is also provided so :code:`make install` (possibly as root or
sudo) will do the steps above.
Running Tests
-------------
::
make check
A GNU makefile has been added to smooth over setting running the right
command, and running tests from fastest to slowest.
If you have remake_ installed, you can see the list of all tasks
including tests via :code:`remake --tasks`
Usage
-----
Run
::
$ uncompyle6 *compiled-python-file-pyc-or-pyo*
For usage help:
::
$ uncompyle6 -h
Verification
------------
In older versions of Python it was possible to verify bytecode by
decompiling bytecode, and then compiling using the Python interpreter
for that bytecode version. Having done this, the bytecode produced
could be compared with the original bytecode. However as Python's code
generation got better, this no longer was feasible.
If you want Python syntax verification of the correctness of the
decompilation process, add the :code:`--syntax-verify` option. However since
Python syntax changes, you should use this option if the bytecode is
the right bytecode for the Python interpreter that will be checking
the syntax.
You can also cross compare the results with another version of
`uncompyle6` since there are sometimes regressions in decompiling
specific bytecode as the overall quality improves.
For Python 3.7 and 3.8, the code in decompyle3_ is generally
better.
Or try specific another python decompiler like uncompyle2_, unpyc37_,
or pycdc_. Since the later two work differently, bugs here often
aren't in that, and vice versa.
There is an interesting class of these programs that is readily
available give stronger verification: those programs that when run
test themselves. Our test suite includes these.
And Python comes with another a set of programs like this: its test
suite for the standard library. We have some code in :code:`test/stdlib` to
facilitate this kind of checking too.
Known Bugs/Restrictions
-----------------------
The biggest known and possibly fixable (but hard) problem has to do
with handling control flow. (Python has probably the most diverse and
screwy set of compound statements I've ever seen; there
are "else" clauses on loops and try blocks that I suspect many
programmers don't know about.)
All of the Python decompilers that I have looked at have problems
decompiling Python's control flow. In some cases we can detect an
erroneous decompilation and report that.
Python support is pretty good for Python 2
On the lower end of Python versions, decompilation seems pretty good although
we don't have any automated testing in place for Python's distributed tests.
Also, we don't have a Python interpreter for versions 1.6, and 2.0.
In the Python 3 series, Python support is strongest around 3.4 or
3.3 and drops off as you move further away from those versions. Python
3.0 is weird in that it in some ways resembles 2.6 more than it does
3.1 or 2.7. Python 3.6 changes things drastically by using word codes
rather than byte codes. As a result, the jump offset field in a jump
instruction argument has been reduced. This makes the :code:`EXTENDED_ARG`
instructions are now more prevalent in jump instruction; previously
they had been rare. Perhaps to compensate for the additional
:code:`EXTENDED_ARG` instructions, additional jump optimization has been
added. So in sum handling control flow by ad hoc means as is currently
done is worse.
Between Python 3.5, 3.6, 3.7 there have been major changes to the
:code:`MAKE_FUNCTION` and :code:`CALL_FUNCTION` instructions.
Python 3.8 removes :code:`SETUP_LOOP`, :code:`SETUP_EXCEPT`,
:code:`BREAK_LOOP`, and :code:`CONTINUE_LOOP`, instructions which may
make control-flow detection harder, lacking the more sophisticated
control-flow analysis that is planned. We'll see.
Currently not all Python magic numbers are supported. Specifically in
some versions of Python, notably Python 3.6, the magic number has
changes several times within a version.
**We support only released versions, not candidate versions.** Note
however that the magic of a released version is usually the same as
the *last* candidate version prior to release.
There are also customized Python interpreters, notably Dropbox,
which use their own magic and encrypt bytecode. With the exception of
the Dropbox's old Python 2.5 interpreter this kind of thing is not
handled.
We also don't handle PJOrion_ or otherwise obfuscated code. For
PJOrion try: PJOrion Deobfuscator_ to unscramble the bytecode to get
valid bytecode before trying this tool; pydecipher_ might help with that.
This program can't decompile Microsoft Windows EXE files created by
Py2EXE_, although we can probably decompile the code after you extract
the bytecode properly. `Pydeinstaller <https://github.com/charles-dyfis-net/pydeinstaller>`_ may help with unpacking Pyinstaller bundlers.
Handling pathologically long lists of expressions or statements is
slow. We don't handle Cython_ or MicroPython which don't use bytecode.
There are numerous bugs in decompilation. And that's true for every
other CPython decompiler I have encountered, even the ones that
claimed to be "perfect" on some particular version like 2.4.
As Python progresses decompilation also gets harder because the
compilation is more sophisticated and the language itself is more
sophisticated. I suspect that attempts there will be fewer ad-hoc
attempts like unpyc37_ (which is based on a 3.3 decompiler) simply
because it is harder to do so. The good news, at least from my
standpoint, is that I think I understand what's needed to address the
problems in a more robust way. But right now until such time as
project is better funded, I do not intend to make any serious effort
to support Python versions 3.8 or 3.9, including bugs that might come
in. I imagine at some point I may be interested in it.
You can easily find bugs by running the tests against the standard
test suite that Python uses to check itself. At any given time, there are
dozens of known problems that are pretty well isolated and that could
be solved if one were to put in the time to do so. The problem is that
there aren't that many people who have been working on bug fixing.
Some of the bugs in 3.7 and 3.8 are simply a matter of back-porting
the fixes in decompyle3. Volunteers are welcome to do so.
You may run across a bug, that you want to report. Please do so after
reading `How to report a bug
<https://github.com/rocky/python-uncompyle6/blob/master/HOW-TO-REPORT-A-BUG.md>`_ and
follow the `instructions when opening an issue <https://github.com/rocky/python-uncompyle6/issues/new?assignees=&labels=&template=bug-report.md>`_.
Be aware that it might not get my attention for a while. If you
sponsor or support the project in some way, I'll prioritize your
issues above the queue of other things I might be doing instead.
See Also
--------
* https://github.com/rocky/python-decompile3 : Much smaller and more modern code, focusing on 3.7 and 3.8. Changes in that will get migrated back here.
* https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique than what is used here. Currently unmaintained.
* https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Includes some fixes like supporting function annotations. Currently unmaintained.
* https://github.com/wibiti/uncompyle2 : supports Python 2.7 only, but does that fairly well. There are situations where :code:`uncompyle6` results are incorrect while :code:`uncompyle2` results are not, but more often uncompyle6 is correct when uncompyle2 is not. Because :code:`uncompyle6` adheres to accuracy over idiomatic Python, :code:`uncompyle2` can produce more natural-looking code when it is correct. Currently :code:`uncompyle2` is lightly maintained. See its issue `tracker <https://github.com/wibiti/uncompyle2/issues>`_ for more details.
* `How to report a bug <https://github.com/rocky/python-uncompyle6/blob/master/HOW-TO-REPORT-A-BUG.md>`_
* The HISTORY_ file.
* https://github.com/rocky/python-xdis : Cross Python version disassembler
* https://github.com/rocky/python-xasm : Cross Python version assembler
* https://github.com/rocky/python-uncompyle6/wiki : Wiki Documents which describe the code and aspects of it in more detail
* https://github.com/zrax/pycdc : The README for this C++ code says it aims to support all versions of Python. You can aim your slign shot for the moon too, but I doubt you are going to hit it. This code is best for Python versions around 2.7 and 3.3 when the code was initially developed. Accuracy for current versions of Python3 and early versions of Python is lacking. Without major effort, it is unlikely it can be made to support current Python 3. See its `issue tracker <https://github.com/zrax/pycdc/issues>`_ for details. Currently lightly maintained.
.. _Cython: https://en.wikipedia.org/wiki/Cython
.. _trepan: https://pypi.python.org/pypi/trepan3k
.. _compiler: https://github.com/rocky/python-uncompyle6/wiki/How-does-this-code-work%3F
.. _HISTORY: https://github.com/rocky/python-uncompyle6/blob/master/HISTORY.md
.. _report_bug: https://github.com/rocky/python-uncompyle6/blob/master/HOW-TO-REPORT-A-BUG.md
.. _debuggers: https://pypi.python.org/pypi/trepan3k
.. _remake: https://bashdb.sf.net/remake
.. _pycdc: https://github.com/zrax/pycdc
.. _decompyle3: https://github.com/rocky/python-decompile3
.. _uncompyle2: https://github.com/wibiti/uncompyle2
.. _unpyc37: https://github.com/andrew-tavera/unpyc37
.. _this: https://github.com/rocky/python-uncompyle6/wiki/Deparsing-technology-and-its-use-in-exact-location-reporting
.. |buildstatus| image:: https://travis-ci.org/rocky/python-uncompyle6.svg
:target: https://travis-ci.org/rocky/python-uncompyle6
.. |packagestatus| image:: https://repology.org/badge/vertical-allrepos/python:uncompyle6.svg
:target: https://repology.org/project/python:uncompyle6/versions
.. _PJOrion: http://www.koreanrandom.com/forum/topic/15280-pjorion-%D1%80%D0%B5%D0%B4%D0%B0%D0%BA%D1%82%D0%B8%D1%80%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5-%D0%BA%D0%BE%D0%BC%D0%BF%D0%B8%D0%BB%D1%8F%D1%86%D0%B8%D1%8F-%D0%B4%D0%B5%D0%BA%D0%BE%D0%BC%D0%BF%D0%B8%D0%BB%D1%8F%D1%86%D0%B8%D1%8F-%D0%BE%D0%B1%D1%84
.. _pydecipher: https://github.com/mitre/pydecipher
.. _Deobfuscator: https://github.com/extremecoders-re/PjOrion-Deobfuscator
.. _Py2EXE: https://en.wikipedia.org/wiki/Py2exe
.. |Supported Python Versions| image:: https://img.shields.io/pypi/pyversions/uncompyle6.svg
.. |Latest Version| image:: https://badge.fury.io/py/uncompyle6.svg
:target: https://badge.fury.io/py/uncompyle6
.. |Pypi Installs| image:: https://pepy.tech/badge/uncompyle6/month
Platform: uncompyle6
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.4
Classifier: Programming Language :: Python :: 2.5
Classifier: Programming Language :: Python :: 2.6
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.0
Classifier: Programming Language :: Python :: 3.1
Classifier: Programming Language :: Python :: 3.2
Classifier: Programming Language :: Python :: 3.3
Classifier: Programming Language :: Python :: 3.4
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Topic :: Software Development :: Debuggers
Classifier: Topic :: Software Development :: Libraries :: Python Modules
License: MIT
Description: UNKNOWN
Platform: UNKNOWN

View File

@@ -1,124 +1,71 @@
|buildstatus| |Pypi Installs| |Latest Version| |Supported Python Versions|
|packagestatus|
.. contents::
|buildstatus| |Supported Python Versions|
uncompyle6
==========
A native Python cross-version decompiler and fragment decompiler.
The successor to decompyle, uncompyle, and uncompyle2.
A native Python cross-version Decompiler and Fragment Decompiler.
Follows in the tradition of decompyle, uncompyle, and uncompyle2.
Introduction
------------
*uncompyle6* translates Python bytecode back into equivalent Python
source code. It accepts bytecodes from Python version 1.0 to version
3.8, spanning over 24 years of Python releases. We include Dropbox's
Python 2.5 bytecode and some PyPy bytecodes.
source code. It accepts bytecodes from Python version 2.1 to 3.6 or
so, including PyPy bytecode and Dropbox's Python 2.5 bytecode.
Why this?
---------
Ok, I'll say it: this software is amazing. It is more than your
normal hacky decompiler. Using compiler_ technology, the program
creates a parse tree of the program from the instructions; nodes at
the upper levels that look a little like what might come from a Python
AST. So we can really classify and understand what's going on in
sections of Python bytecode.
There were a number of decompyle, uncompile, uncompyle2, uncompyle3
forks around. All of them came basically from the same code base, and
almost all of them no were no longer actively maintained. Only one
handled Python 3, and even there, only 3.2 or 3.3 depending on which
code is used. This code pulls these together and moves forward. It
also addresses a number of open issues in the previous forks.
Building on this, another thing that makes this different from other
CPython bytecode decompilers is the ability to deparse just
*fragments* of source code and give source-code information around a
given bytecode offset.
What makes this different from other CPython bytecode decompilers?: its
ability to deparse just fragments and give source-code information
around a given bytecode offset.
I use the tree fragments to deparse fragments of code *at run time*
inside my trepan_ debuggers_. For that, bytecode offsets are recorded
and associated with fragments of the source code. This purpose,
although compatible with the original intention, is yet a little bit
different. See this_ for more information.
I use this to deparse fragments of code inside my trepan_
debuggers_. For that, I need to record text fragments for all
bytecode offsets (of interest). This purpose although largely
compatible with the original intention is yet a little bit different.
See this_ for more information.
Python fragment deparsing given an instruction offset is useful in
showing stack traces and can be incorporated into any program that
wants to show a location in more detail than just a line number at
runtime. This code can be also used when source-code information does
not exist and there is just bytecode. Again, my debuggers make use of
this.
There were (and still are) a number of decompyle, uncompyle,
uncompyle2, uncompyle3 forks around. Many of them come basically from
the same code base, and (almost?) all of them are no longer actively
maintained. One was really good at decompiling Python 1.5-2.3, another
really good at Python 2.7, but that only. Another handles Python 3.2
only; another patched that and handled only 3.3. You get the
idea. This code pulls all of these forks together and *moves
forward*. There is some serious refactoring and cleanup in this code
base over those old forks. Even more experimental refactoring is going
on in decompyle3_.
This demonstrably does the best in decompiling Python across all
Python versions. And even when there is another project that only
provides decompilation for subset of Python versions, we generally do
demonstrably better for those as well.
How can we tell? By taking Python bytecode that comes distributed with
that version of Python and decompiling these. Among those that
successfully decompile, we can then make sure the resulting programs
are syntactically correct by running the Python interpreter for that
bytecode version. Finally, in cases where the program has a test for
itself, we can run the check on the decompiled code.
We use an automated processes to find bugs. In the issue trackers for
other decompilers, you will find a number of bugs we've found along
the way. Very few to none of them are fixed in the other decompilers.
The idea of Python fragment deparsing given an instruction offset can
be used in showing stack traces or any program that wants to show a
location in more detail than just a line number. It can be also used
when source-code information does not exist and there is just bytecode
information.
Requirements
------------
The code in the git repository can be run from Python 2.4 to the
latest Python version, with the exception of Python 3.0 through
3.2. Volunteers are welcome to address these deficiencies if there a
desire to do so.
The way it does this though is by segregating consecutive Python versions into
git branches:
master
Python 3.6 and up (uses type annotations)
python-3.3-to-3.5
Python 3.3 through 3.5 (Generic Python 3)
python-2.4
Python 2.4 through 2.7 (Generic Python 2)
PyPy 3-2.4 and later works as well.
The bytecode files it can read have been tested on Python
bytecodes from versions 1.4, 2.1-2.7, and 3.0-3.8 and later PyPy
versions.
This project requires Python 2.6 or later, PyPy 3-2.4, or PyPy-5.0.1.
Python versions 2.3-2.7 are supported in the python-2.4 branch.
The bytecode files it can read has been tested on Python bytecodes from
versions 2.1-2.7, and 3.2-3.6 and the above-mentioned PyPy versions.
Installation
------------
You can install from PyPI using the name ``uncompyle6``::
This uses setup.py, so it follows the standard Python routine:
pip install uncompyle6
::
pip install -r requirements.txt
pip install -r requirements-dev.txt
python setup.py install # may need sudo
# or if you have pyenv:
python setup.py develop
To install from source code, this project uses setup.py, so it follows the standard Python routine::
$ pip install -e . # set up to run from source tree
or::
$ python setup.py install # may need sudo
A GNU Makefile is also provided so :code:`make install` (possibly as root or
A GNU makefile is also provided so :code:`make install` (possibly as root or
sudo) will do the steps above.
Running Tests
-------------
Testing
-------
::
@@ -146,176 +93,79 @@ For usage help:
$ uncompyle6 -h
Verification
------------
In older versions of Python it was possible to verify bytecode by
decompiling bytecode, and then compiling using the Python interpreter
for that bytecode version. Having done this, the bytecode produced
could be compared with the original bytecode. However as Python's code
generation got better, this no longer was feasible.
If you want Python syntax verification of the correctness of the
decompilation process, add the :code:`--syntax-verify` option. However since
Python syntax changes, you should use this option if the bytecode is
the right bytecode for the Python interpreter that will be checking
the syntax.
You can also cross compare the results with another version of
*uncompyle6* since there are sometimes regressions in decompiling
specific bytecode as the overall quality improves.
For Python 3.7 and 3.8, the code in decompyle3_ is generally
better.
Or try specific another python decompiler like uncompyle2_, unpyc37_,
or pycdc_. Since the later two work differently, bugs here often
aren't in that, and vice versa.
There is an interesting class of these programs that is readily
available give stronger verification: those programs that when run
test themselves. Our test suite includes these.
And Python comes with another a set of programs like this: its test
suite for the standard library. We have some code in :code:`test/stdlib` to
facilitate this kind of checking too.
Known Bugs/Restrictions
-----------------------
The biggest known and possibly fixable (but hard) problem has to do
with handling control flow. (Python has probably the most diverse and
screwy set of compound statements I've ever seen; there
are "else" clauses on loops and try blocks that I suspect many
programmers don't know about.)
with handling control flow. All of the Python decompilers I have looked
at have the same problem. In some cases we can detect an erroneous
decompilation and report that.
All of the Python decompilers that I have looked at have problems
decompiling Python's control flow. In some cases we can detect an
erroneous decompilation and report that.
About 90% of the decompilation of Python standard library packages in
Python 2.7.12 verifies correctly. Over 99% of Python 2.7 and 3.3-3.5
"weakly" verify. Python 2.6 drops down to 96% weakly verifying.
Other versions drop off in quality too.
Python support is pretty good for Python 2
*Verification* is the process of decompiling bytecode, compiling with
a Python for that bytecode version, and then comparing the bytecode
produced by the decompiled/compiled program. Some allowance is made
for inessential differences. But other semantically equivalent
differences are not caught. For example ``1 and 0`` is decompiled to
the equivalent ``0``; remnants of the first true evaluation (1) is
lost when Python compiles this. When Python next compiles ``0`` the
resulting code is simpler.
On the lower end of Python versions, decompilation seems pretty good although
we don't have any automated testing in place for Python's distributed tests.
Also, we don't have a Python interpreter for versions 1.6, and 2.0.
*Weak Verification*
on the other hand doesn't check bytecode for equivalence but does
check to see if the resulting decompiled source is a valid Python
program by running the Python interpreter. Because the Python language
has changed so much, for best results you should use the same Python
Version in checking as used in the bytecode.
In the Python 3 series, Python support is strongest around 3.4 or
3.3 and drops off as you move further away from those versions. Python
3.0 is weird in that it in some ways resembles 2.6 more than it does
3.1 or 2.7. Python 3.6 changes things drastically by using word codes
rather than byte codes. As a result, the jump offset field in a jump
instruction argument has been reduced. This makes the :code:`EXTENDED_ARG`
instructions are now more prevalent in jump instruction; previously
they had been rare. Perhaps to compensate for the additional
:code:`EXTENDED_ARG` instructions, additional jump optimization has been
added. So in sum handling control flow by ad hoc means as is currently
done is worse.
Later distributions average about 200 files. There is some work to do
on the lower end Python versions which is more difficult for us to
handle since we don't have a Python interpreter for versions 1.5, 1.6,
and 2.0.
Between Python 3.5, 3.6, 3.7 there have been major changes to the
:code:`MAKE_FUNCTION` and :code:`CALL_FUNCTION` instructions.
Python 3.8 removes :code:`SETUP_LOOP`, :code:`SETUP_EXCEPT`,
:code:`BREAK_LOOP`, and :code:`CONTINUE_LOOP`, instructions which may
make control-flow detection harder, lacking the more sophisticated
control-flow analysis that is planned. We'll see.
Python 3.0 support is weak; Python 3.5 largely works, but still has
some bugs in it. Python 3.6 changes things drastically by using word
codes rather than byte codes. That has been addressed, but then it also
changes function call opcodes and its semantics.
Currently not all Python magic numbers are supported. Specifically in
some versions of Python, notably Python 3.6, the magic number has
changes several times within a version.
**We support only released versions, not candidate versions.** Note
however that the magic of a released version is usually the same as
the *last* candidate version prior to release.
There are also customized Python interpreters, notably Dropbox,
which use their own magic and encrypt bytecode. With the exception of
changes several times within a version. We support only the released
magic. There are also customized Python interpreters, notably Dropbox,
which use their own magic and encrypt bytcode. With the exception of
the Dropbox's old Python 2.5 interpreter this kind of thing is not
handled.
We also don't handle PJOrion_ or otherwise obfuscated code. For
PJOrion try: PJOrion Deobfuscator_ to unscramble the bytecode to get
valid bytecode before trying this tool; pydecipher_ might help with that.
We also don't handle PJOrion_ obfuscated code. For that try: PJOrion
Deobfuscator_ to unscramble the bytecode to get valid bytecode before
trying this tool.
This program can't decompile Microsoft Windows EXE files created by
Py2EXE_, although we can probably decompile the code after you extract
the bytecode properly. `Pydeinstaller <https://github.com/charles-dyfis-net/pydeinstaller>`_ may help with unpacking Pyinstaller bundlers.
Handling pathologically long lists of expressions or statements is
slow. We don't handle Cython_ or MicroPython which don't use bytecode.
There are numerous bugs in decompilation. And that's true for every
other CPython decompiler I have encountered, even the ones that
claimed to be "perfect" on some particular version like 2.4.
As Python progresses decompilation also gets harder because the
compilation is more sophisticated and the language itself is more
sophisticated. I suspect that attempts there will be fewer ad-hoc
attempts like unpyc37_ (which is based on a 3.3 decompiler) simply
because it is harder to do so. The good news, at least from my
standpoint, is that I think I understand what's needed to address the
problems in a more robust way. But right now until such time as
project is better funded, I do not intend to make any serious effort
to support Python versions 3.8 or 3.9, including bugs that might come
in. I imagine at some point I may be interested in it.
You can easily find bugs by running the tests against the standard
test suite that Python uses to check itself. At any given time, there are
dozens of known problems that are pretty well isolated and that could
be solved if one were to put in the time to do so. The problem is that
there aren't that many people who have been working on bug fixing.
Some of the bugs in 3.7 and 3.8 are simply a matter of back-porting
the fixes in *decompyle3*. Any volunteers?
You may run across a bug, that you want to report. Please do so after
reading `How to report a bug
<https://github.com/rocky/python-uncompyle6/blob/master/HOW-TO-REPORT-A-BUG.md>`_ and
follow the `instructions when opening an issue <https://github.com/rocky/python-uncompyle6/issues/new?assignees=&labels=&template=bug-report.md>`_.
Be aware that it might not get my attention for a while. If you
sponsor or support the project in some way, I'll prioritize your
issues above the queue of other things I might be doing instead. In
rare situtations, I can do a hand decompilation of bytecode for a fee.
However this is expansive, usually beyond what most people are willing
to spend.
There is lots to do, so please dig in and help.
See Also
--------
* https://rocky.github.io/blackhat-asia-2024-additional/all-notes-print.html : How to Read and Write a High-Level Bytecode Decompiler: ``uncompyle6`` ``decompyle3`` -- BlackHat 2024 Asia (`video <https://www.youtube.com/watch?v=NA77SFncppE>`_. A big thanks to the Organizers and Reviewers for letting me speak. This kind of thing encourages me to work on projects like this.
* https://github.com/rocky/python-decompile3 : Much smaller and more modern code, focusing on 3.7 and 3.8. Changes in that will get migrated back here.
* https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique than what is used here. Currently unmaintained.
* https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Includes some fixes like supporting function annotations. Currently unmaintained.
* https://github.com/wibiti/uncompyle2 : supports Python 2.7 only, but does that fairly well. There are situations where :code:`uncompyle6` results are incorrect while :code:`uncompyle2` results are not, but more often uncompyle6 is correct when uncompyle2 is not. Because :code:`uncompyle6` adheres to accuracy over idiomatic Python, :code:`uncompyle2` can produce more natural-looking code when it is correct. Currently :code:`uncompyle2` is lightly maintained. See its issue `tracker <https://github.com/wibiti/uncompyle2/issues>`_ for more details.
* `How to report a bug <https://github.com/rocky/python-uncompyle6/blob/master/HOW-TO-REPORT-A-BUG.md>`_
* https://github.com/zrax/pycdc : supports all versions of Python and is written in C++
* https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique what is used here.
* https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Include some fixes like supporting function annotations
* The HISTORY_ file.
* https://github.com/rocky/python-xdis : Cross Python version disassembler
* https://github.com/rocky/python-xasm : Cross Python version assembler
* https://github.com/rocky/python-uncompyle6/wiki : Wiki Documents which describe the code and aspects of it in more detail
* https://github.com/zrax/pycdc : The README for this C++ code says it aims to support all versions of Python. You can aim your slign shot for the moon too, but I doubt you are going to hit it. This code is best for Python versions around 2.7 and 3.3 when the code was initially developed. Accuracy for current versions of Python3 and early versions of Python is lacking. Without major effort, it is unlikely it can be made to support current Python 3. See its `issue tracker <https://github.com/zrax/pycdc/issues>`_ for details. Currently lightly maintained.
.. _Cython: https://en.wikipedia.org/wiki/Cython
.. _trepan: https://pypi.python.org/pypi/trepan3k
.. _compiler: https://github.com/rocky/python-uncompyle6/wiki/How-does-this-code-work%3F
.. |downloads| image:: https://img.shields.io/pypi/dd/uncompyle6.svg
.. _trepan: https://pypi.python.org/pypi/trepan
.. _HISTORY: https://github.com/rocky/python-uncompyle6/blob/master/HISTORY.md
.. _report_bug: https://github.com/rocky/python-uncompyle6/blob/master/HOW-TO-REPORT-A-BUG.md
.. _debuggers: https://pypi.python.org/pypi/trepan3k
.. _remake: https://bashdb.sf.net/remake
.. _pycdc: https://github.com/zrax/pycdc
.. _decompyle3: https://github.com/rocky/python-decompile3
.. _uncompyle2: https://github.com/wibiti/uncompyle2
.. _unpyc37: https://github.com/andrew-tavera/unpyc37
.. _this: https://github.com/rocky/python-uncompyle6/wiki/Deparsing-technology-and-its-use-in-exact-location-reporting
.. |buildstatus| image:: https://circleci.com/gh/rocky/python-uncompyle6.svg?style=svg
:target: https://app.circleci.com/pipelines/github/rocky/python-uncompyle6
.. |packagestatus| image:: https://repology.org/badge/vertical-allrepos/python:uncompyle6.svg
:target: https://repology.org/project/python:uncompyle6/versions
.. _PJOrion: http://www.koreanrandom.com/forum/topic/15280-pjorion-%D1%80%D0%B5%D0%B4%D0%B0%D0%BA%D1%82%D0%B8%D1%80%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5-%D0%BA%D0%BE%D0%BC%D0%BF%D0%B8%D0%BB%D1%8F%D1%86%D0%B8%D1%8F-%D0%B4%D0%B5%D0%BA%D0%BE%D0%BC%D0%BF%D0%B8%D0%BB%D1%8F%D1%86%D0%B8%D1%8F-%D0%BE%D0%B1%D1%84
.. _pydecipher: https://github.com/mitre/pydecipher
.. _Deobfuscator: https://github.com/extremecoders-re/PjOrion-Deobfuscator
.. _Py2EXE: https://en.wikipedia.org/wiki/Py2exe
.. |buildstatus| image:: https://travis-ci.org/rocky/python-uncompyle6.svg
:target: https://travis-ci.org/rocky/python-uncompyle6
.. |Supported Python Versions| image:: https://img.shields.io/pypi/pyversions/uncompyle6.svg
.. |Latest Version| image:: https://badge.fury.io/py/uncompyle6.svg
:target: https://badge.fury.io/py/uncompyle6
.. |Pypi Installs| image:: https://pepy.tech/badge/uncompyle6/month
:target: https://pypi.python.org/pypi/uncompyle6/
.. _PJOrion: http://www.koreanrandom.com/forum/topic/15280-pjorion-%D1%80%D0%B5%D0%B4%D0%B0%D0%BA%D1%82%D0%B8%D1%80%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5-%D0%BA%D0%BE%D0%BC%D0%BF%D0%B8%D0%BB%D1%8F%D1%86%D0%B8%D1%8F-%D0%B4%D0%B5%D0%BA%D0%BE%D0%BC%D0%BF%D0%B8%D0%BB%D1%8F%D1%86%D0%B8%D1%8F-%D0%BE%D0%B1%D1%84
.. _Deobfuscator: https://github.com/extremecoders-re/PjOrion-Deobfuscator

View File

View File

@@ -1,17 +1,3 @@
# Copyright (C) 2018, 2020-2021 2024 Rocky Bernstein <rocky@gnu.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""uncompyle6 packaging information"""
# To the extent possible we make this file look more like a
@@ -21,89 +7,59 @@
# less elegant than having it here with reduced code, albeit there
# still is some room for improvement.
# Python-version | package | last-version |
# -----------------------------------------
# 2.5 | pip | 1.1 |
# 2.6 | pip | 1.5.6 |
# 2.7 | pip | 19.2.3 |
# 2.7 | pip | 1.2.1 |
# 3.1 | pip | 1.5.6 |
# 3.2 | pip | 7.1.2 |
# 3.3 | pip | 10.0.1 |
# 3.4 | pip | 19.1.1 |
import os.path as osp
# Things that change more often go here.
copyright = """
Copyright (C) 2015-2021, 2024 Rocky Bernstein <rb@dustyfeet.com>.
copyright = """
Copyright (C) 2015, 2016 Rocky Bernstein <rb@dustyfeet.com>.
"""
classifiers = [
"Development Status :: 5 - Production/Stable",
"Intended Audience :: Developers",
"License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
"Operating System :: OS Independent",
"Programming Language :: Python",
"Programming Language :: Python :: 2",
"Programming Language :: Python :: 2.4",
"Programming Language :: Python :: 2.5",
"Programming Language :: Python :: 2.6",
"Programming Language :: Python :: 2.7",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.0",
"Programming Language :: Python :: 3.1",
"Programming Language :: Python :: 3.2",
"Programming Language :: Python :: 3.3",
"Programming Language :: Python :: 3.4",
"Programming Language :: Python :: 3.5",
"Programming Language :: Python :: 3.6",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: Implementation :: PyPy",
"Topic :: Software Development :: Debuggers",
"Topic :: Software Development :: Libraries :: Python Modules",
]
classifiers = ['Development Status :: 5 - Production/Stable',
'Intended Audience :: Developers',
'Operating System :: OS Independent',
'Programming Language :: Python',
'Programming Language :: Python :: 2.4',
'Programming Language :: Python :: 2.5',
'Programming Language :: Python :: 2.6',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3.3',
'Programming Language :: Python :: 3.4',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
'Topic :: Software Development :: Debuggers',
'Topic :: Software Development :: Libraries :: Python Modules',
]
# The rest in alphabetic order
author = "Rocky Bernstein, Hartmut Goebel, John Aycock, and others"
author_email = "rb@dustyfeet.com"
entry_points = {
"console_scripts": [
"uncompyle6=uncompyle6.bin.uncompile:main_bin",
"pydisassemble=uncompyle6.bin.pydisassemble:main",
]
}
ftp_url = None
install_requires = ["click", "spark-parser >= 1.8.9, < 1.9.2", "xdis >= 6.1.1, < 6.2.0"]
license = "GPL3"
mailing_list = "python-debugger@googlegroups.com"
modname = "uncompyle6"
py_modules = None
short_desc = "Python cross-version byte-code decompiler"
web = "https://github.com/rocky/python-uncompyle6/"
author = "Rocky Bernstein, Hartmut Goebel, John Aycock, and others"
author_email = "rb@dustyfeet.com"
entry_points={
'console_scripts': [
'uncompyle6=uncompyle6.bin.uncompile:main_bin',
'pydisassemble=uncompyle6.bin.pydisassemble:main',
]}
ftp_url = None
install_requires = ['spark-parser >= 1.5.1, < 1.6.0',
'xdis >= 3.2.3, < 3.3.0']
license = 'MIT'
mailing_list = 'python-debugger@googlegroups.com'
modname = 'uncompyle6'
py_modules = None
short_desc = 'Python cross-version byte-code deparser'
web = 'https://github.com/rocky/python-uncompyle6/'
# tracebacks in zip files are funky and not debuggable
zip_safe = True
import os.path
def get_srcdir():
filename = osp.normcase(osp.dirname(osp.abspath(__file__)))
return osp.realpath(filename)
filename = os.path.normcase(os.path.dirname(os.path.abspath(__file__)))
return os.path.realpath(filename)
srcdir = get_srcdir()
def read(*rnames):
return open(osp.join(srcdir, *rnames)).read()
return open(os.path.join(srcdir, *rnames)).read()
# Get info from files; set: long_description and VERSION
long_description = read("README.rst") + "\n"
exec(read("uncompyle6/version.py"))
long_description = ( read("README.rst") + '\n' )
exec(read('uncompyle6/version.py'))

View File

@@ -1 +0,0 @@
/.python-version

View File

@@ -1,22 +0,0 @@
Making a release is a somewhat tedious process so I've automated it a little
Here are tools that I, rocky, use to check and build a distribution.
They are customized to my environment:
- I use pyenv to various Python versions installed
- I have git repos for xdis, and spark parser at the same level as uncompyle6
There may be other rocky-specific things that need customization.
how-to-make-a-release.md has overall how I make a release
Since this project uses python over a wide variety of release, some versions
of projects that should be used for specific Python versions
for 3.2.6:
pytest==2.9.2
for 3.1.5
pytset==2.1.0
py=1.8.0 and comment out line 10 of _builtin.py # callable = callable
six==1.10.0

View File

@@ -1,34 +0,0 @@
#!/bin/bash
# Run tests over all Python versions in branch python-2.4-2.7
set -e
function finish {
cd $owd
}
owd=$(pwd)
trap finish EXIT
cd $(dirname ${BASH_SOURCE[0]})
if ! source ./pyenv-2.4-2.7-versions ; then
exit $?
fi
if ! source ./setup-python-2.4.sh ; then
rc=$?
finish
exit $rc
fi
cd ..
for version in $PYVERSIONS; do
echo --- $version ---
if ! pyenv local $version ; then
exit $?
fi
make clean && python setup.py develop
if ! make check ; then
finish
rc=$?
exit $?
fi
echo === $version ===
done
finish

View File

@@ -1,31 +0,0 @@
#!/bin/bash
# Run tests over all Python versions in branch python-3.0-3.2
set -e
function finish {
cd $owd
}
owd=$(pwd)
trap finish EXIT
cd $(dirname ${BASH_SOURCE[0]})
if ! source ./pyenv-3.0-3.2-versions ; then
exit $?
fi
if ! source ./setup-python-3.0.sh ; then
exit $?
fi
cd ..
for version in $PYVERSIONS; do
echo --- $version ---
if ! pyenv local $version ; then
exit $?
fi
make clean && python setup.py develop
if ! make check ; then
exit $?
fi
echo === $version ===
done
finish

View File

@@ -1,30 +0,0 @@
#!/bin/bash
# Run tests over all Python versions in branch python-3.3-3.5
set -e
function finish {
cd $owd
}
owd=$(pwd)
trap finish EXIT
cd $(dirname ${BASH_SOURCE[0]})
if ! source ./pyenv-3.3-3.5-versions ; then
exit $?
fi
if ! source ./setup-python-3.3.sh ; then
exit $?
fi
cd ..
for version in $PYVERSIONS; do
echo --- $version ---
if ! pyenv local $version ; then
exit $?
fi
make clean && python setup.py develop
if ! make check ; then
exit $?
fi
echo === $version ===
done
finish

View File

@@ -1,29 +0,0 @@
#!/bin/bash
function finish {
cd $owd
}
# FIXME put some of the below in a common routine
owd=$(pwd)
trap finish EXIT
cd $(dirname ${BASH_SOURCE[0]})
if ! source ./pyenv-newest-versions ; then
exit $?
fi
if ! source ./setup-master.sh ; then
exit $?
fi
cd ..
for version in $PYVERSIONS; do
echo --- $version ---
if ! pyenv local $version ; then
exit $?
fi
make clean && pip install -e .
if ! make check-short; then
exit $?
fi
echo === $version ===
done
make check

View File

@@ -1,21 +0,0 @@
# Common checkout routine
export PATH=$HOME/.pyenv/bin/pyenv:$PATH
bs=${BASH_SOURCE[0]}
mydir=$(dirname $bs)
fulldir=$(readlink -f $mydir)
function setup_version {
local repo=$1
version=$2
echo Running setup $version on $repo ...
(cd ../$repo && . ./admin-tools/setup-${version}.sh)
return $?
}
function checkout_finish {
branch=$1
cd $uncompyle6_owd
git checkout $branch && pyenv local $PYTHON_VERSION && git pull
rc=$?
return $rc
}

View File

@@ -1,100 +0,0 @@
<!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc -->
**Table of Contents**
- [Get latest sources:](#get-latest-sources)
- [Change version in uncompyle6/version.py:](#change-version-in-uncompyle6versionpy)
- [Update ChangeLog:](#update-changelog)
- [Update NEWS.md from ChangeLog:](#update-newsmd-from-changelog)
- [Make sure pyenv is running and check newer versions](#make-sure-pyenv-is-running-and-check-newer-versions)
- [Switch to python-2.4, sync that up and build that first since it creates a tarball which we don't want.](#switch-to-python-24-sync-that-up-and-build-that-first-since-it-creates-a-tarball-which-we-dont-want)
- [Check against older versions](#check-against-older-versions)
- [Make packages and tag](#make-packages-and-tag)
- [Check package on github](#check-package-on-github)
- [Release on Github](#release-on-github)
- [Get onto PyPI](#get-onto-pypi)
- [Update tags:](#update-tags)
<!-- markdown-toc end -->
# Get latest sources:
git pull
# Change version in uncompyle6/version.py:
$ emacs uncompyle6/version.py
$ source uncompyle6/version.py
$ echo $VERSION
$ git commit -m"Get ready for release $VERSION" .
# Update ChangeLog:
$ make ChangeLog
# Update NEWS.md from ChangeLog:
$ emacs NEWS.md
$ make check
$ git commit --amend .
$ git push # get CI testing going early
# Make sure pyenv is running and check newer versions
$ admin-tools/check-newer-versions.sh
# Switch to python-2.4, sync that up and build that first since it creates a tarball which we don't want.
$ source admin-tools/setup-python-2.4.sh
$ git merge master
# Add and fix merge conflicts
$ git commit
# Check against older versions
$ admin-tools/check-older-versions.sh
# Make packages and tag
$ . ./admin-tools/make-dist-older.sh
$ pyenv local 3.8.5
$ twine check dist/uncompyle6-$VERSION*
$ git tag release-python-2.4-$VERSION
$ ./admin-tools/make-dist-newer.sh
$ twine check dist/uncompyle6-$VERSION*
# Check package on github
$ [[ ! -d /tmp/gittest ]] && mkdir /tmp/gittest; pushd /tmp/gittest
$ pyenv local 3.8.3
$ pip install -e git://github.com/rocky/python-uncompyle6.git#egg=uncompyle6
$ uncompyle6 --help
$ pip uninstall uncompyle6
$ popd
# Release on Github
Goto https://github.com/rocky/python-uncompyle6/releases
Now check the *tagged* release. (Checking the untagged release was previously done).
Todo: turn this into a script in `admin-tools`
$ pushd /tmp/gittest
$ pip install -e git://github.com/rocky/python-uncompyle6.git@$VERSION#egg=uncompyle6
$ uncompyle6 --help
$ pip uninstall uncompyle6
$ popd
# Get onto PyPI
$ twine upload dist/uncompyle6-${VERSION}*
# Update tags:
$ git push --tags
$ git pull --tags
# Move dist files to uploaded
$ mv -v dist/uncompyle6-${VERSION}* dist/uploaded

View File

@@ -1,49 +0,0 @@
#!/bin/bash
PACKAGE=uncompyle6
# FIXME put some of the below in a common routine
function finish {
cd $make_dist_uncompyle6_owd
}
make_dist_uncompyle6_owd=$(pwd)
trap finish EXIT
cd $(dirname ${BASH_SOURCE[0]})
if ! source ./pyenv-2.4-2.7-versions ; then
exit $?
fi
if ! source ./setup-python-2.4.sh ; then
exit $?
fi
cd ..
source $PACKAGE/version.py
echo $__version__
for pyversion in $PYVERSIONS; do
echo --- $pyversion ---
if [[ ${pyversion:0:4} == "pypy" ]] ; then
echo "$pyversion - PyPy does not get special packaging"
continue
fi
if ! pyenv local $pyversion ; then
exit $?
fi
rm -fr build
python setup.py bdist_egg
done
pyenv local 2.7.18
python setup.py bdist_wheel
mv -v dist/${PACKAGE}-$__version__-py2{.py3,}-none-any.whl
# Pypi can only have one source tarball.
# Tarballs can get created from the above setup, so make sure to remove them since we want
# the tarball from master.
tarball=dist/${PACKAGE}-${__version_}_-tar.gz
if [[ -f $tarball ]]; then
rm -v dist/${PACKAGE}-${__version__}-tar.gz
fi
finish

View File

@@ -1,49 +0,0 @@
#!/bin/bash
PACKAGE=uncompyle6
# FIXME put some of the below in a common routine
function finish {
cd $uncompyle6_30_make_dist_owd
}
cd $(dirname ${BASH_SOURCE[0]})
uncompyle6_30_make_dist_owd=$(pwd)
trap finish EXIT
if ! source ./pyenv-3.0-3.2-versions ; then
exit $?
fi
if ! source ./setup-python-3.0.sh ; then
exit $?
fi
cd ..
source $PACKAGE/version.py
echo $__version__
for pyversion in $PYVERSIONS; do
echo --- $pyversion ---
if [[ ${pyversion:0:4} == "pypy" ]] ; then
echo "$pyversion - PyPy does not get special packaging"
continue
fi
if ! pyenv local $pyversion ; then
exit $?
fi
# pip bdist_egg create too-general wheels. So
# we narrow that by moving the generated wheel.
# Pick out first two number of version, e.g. 3.5.1 -> 35
first_two=$(echo $pyversion | cut -d'.' -f 1-2 | sed -e 's/\.//')
rm -fr build
python setup.py bdist_egg bdist_wheel
mv -v dist/${PACKAGE}-$__version__-{py2.py3,py$first_two}-none-any.whl
echo === $pyversion ===
done
python ./setup.py sdist
tarball=dist/${PACKAGE}-${__version__}.tar.gz
if [[ -f $tarball ]]; then
mv -v $tarball dist/${PACKAGE}_31-${__version__}.tar.gz
fi
finish

View File

@@ -1,49 +0,0 @@
#!/bin/bash
PACKAGE=uncompyle6
# FIXME put some of the below in a common routine
function finish {
cd $uncompyle6_33_make_owd
}
cd $(dirname ${BASH_SOURCE[0]})
uncompyle6_33_make_owd=$(pwd)
trap finish EXIT
if ! source ./pyenv-3.3-3.5-versions ; then
exit $?
fi
if ! source ./setup-python-3.3.sh ; then
exit $?
fi
cd ..
source $PACKAGE/version.py
echo $__version__
for pyversion in $PYVERSIONS; do
echo --- $pyversion ---
if [[ ${pyversion:0:4} == "pypy" ]] ; then
echo "$pyversion - PyPy does not get special packaging"
continue
fi
if ! pyenv local $pyversion ; then
exit $?
fi
# pip bdist_egg create too-general wheels. So
# we narrow that by moving the generated wheel.
# Pick out first two number of version, e.g. 3.5.1 -> 35
first_two=$(echo $pyversion | cut -d'.' -f 1-2 | sed -e 's/\.//')
rm -fr build
python setup.py bdist_egg bdist_wheel
mv -v dist/${PACKAGE}-$__version__-{py2.py3,py$first_two}-none-any.whl
echo === $pyversion ===
done
python ./setup.py sdist
tarball=dist/${PACKAGE}-${__version__}.tar.gz
if [[ -f $tarball ]]; then
mv -v $tarball dist/${PACKAGE}_31-${__version__}.tar.gz
fi
finish

View File

@@ -1,44 +0,0 @@
#!/bin/bash
PACKAGE=uncompyle6
# FIXME put some of the below in a common routine
function finish {
cd $make_uncompyle6_newest_owd
}
cd $(dirname ${BASH_SOURCE[0]})
make_uncompyle6_newest_owd=$(pwd)
trap finish EXIT
if ! source ./pyenv-newest-versions ; then
exit $?
fi
if ! source ./setup-master.sh ; then
exit $?
fi
cd ..
source $PACKAGE/version.py
echo $__version__
for pyversion in $PYVERSIONS; do
echo --- $pyversion ---
if [[ ${pyversion:0:4} == "pypy" ]] ; then
echo "$pyversion - PyPy does not get special packaging"
continue
fi
if ! pyenv local $pyversion ; then
exit $?
fi
# pip bdist_egg create too-general wheels. So
# we narrow that by moving the generated wheel.
# Pick out first two number of version, e.g. 3.5.1 -> 35
first_two=$(echo $pyversion | cut -d'.' -f 1-2 | sed -e 's/\.//')
rm -fr build
python setup.py bdist_egg bdist_wheel
mv -v dist/${PACKAGE}-$__version__-{py2.py3,py$first_two}-none-any.whl
done
python ./setup.py sdist
finish

View File

@@ -1,7 +0,0 @@
#/bin/bash
uncompyle6_merge_24_owd=$(pwd)
cd $(dirname ${BASH_SOURCE[0]})
if . ./setup-python-2.4.sh; then
git merge python-3.0-to-3.2
fi
cd $uncompyle6_merge_24_owd

View File

@@ -1,7 +0,0 @@
#/bin/bash
uncompyle6_merge_30_owd=$(pwd)
cd $(dirname ${BASH_SOURCE[0]})
if . ./setup-python-3.0.sh; then
git merge python-3.3-to-3.5
fi
cd $uncompyle6_merge_30_owd

View File

@@ -1,7 +0,0 @@
#/bin/bash
uncompyle6_merge_33_owd=$(pwd)
cd $(dirname ${BASH_SOURCE[0]})
if . ./setup-python-3.3.sh; then
git merge python-3.6-to-3.10
fi
cd $uncompyle6_merge_33_owd

View File

@@ -1,7 +0,0 @@
#/bin/bash
uncompyle6_merge_36_owd=$(pwd)
cd $(dirname ${BASH_SOURCE[0]})
if . ./setup-python-3.6.sh; then
git merge master
fi
cd $uncompyle6_merge_36_owd

View File

@@ -1,15 +0,0 @@
#!/bin/bash
# Use pycdc to run our test/bytecode* test suite
bs=${BASH_SOURCE[0]}
testdir=$(dirname $bs)/../test
fulldir=$(readlink -f $testdir)
cd $fulldir
for dir in bytecode_* ; do
echo ========= $dir ================
cd $fulldir/$dir
for file in *.pyc; do
if ! pycdc $file > /dev/null ; then
echo ----- $dir/$file ------
fi
done
done

View File

@@ -1,9 +0,0 @@
# -*- shell-script -*-
# Sets PYVERSIONS to be all pyenv the oldest versions we have.
# These are not covered (yet) by uncompyle6, although
# some programs do work here.
if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
echo "This script should be *sourced* rather than run directly through bash"
exit 1
fi
export PYVERSIONS='2.1.3 2.2.3 2.3.7'

View File

@@ -1,9 +0,0 @@
# -*- shell-script -*-
# Sets PYVERSIONS to be pyenv versions that
# we can use in the python-2.4-to-2.7 branch.
if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
echo "This script should be *sourced* rather than run directly through bash"
exit 1
fi
export PYVERSIONS='2.4.6 2.5.6 2.6.9 2.7.18'

View File

@@ -1,9 +0,0 @@
# -*- shell-script -*-
# Sets PYVERSIONS to be pyenv versions that
# we can use in the python-2.4 branch.
if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
echo "This script should be *sourced* rather than run directly through bash"
exit 1
fi
export PYVERSIONS='3.0.1 3.1.5 3.2.6'

View File

@@ -1,8 +0,0 @@
# -*- shell-script -*-
# Sets PYVERSIONS to be pyenv versions that
# we can use in the python-3.3-to-3.5 branch.
if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
echo "This script should be *sourced* rather than run directly through bash"
exit 1
fi
export PYVERSIONS=' 3.3.7 3.4.10 3.5.10 '

View File

@@ -1,19 +0,0 @@
# -*- shell-script -*-
# Sets PYVERSIONS to be all pyenv versions we have
if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
echo "This script should be *sourced* rather than run directly through bash"
exit 1
fi
olddir=$(pwd)
mydir=$(dirname ${BASH_SOURCE[0]})
cd $mydir
all=""
for file in pyenv-{olde{st,r},newer}-versions ; do
. $mydir/$file
all="$all $PYVERSIONS"
done
PYVERSIONS="$all"
cd $olddir

View File

@@ -1,8 +0,0 @@
# -*- shell-script -*-
# Sets PYVERSIONS to be pyenv versions that
# we can use in the master branch.
if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
echo "This script should be *sourced* rather than run directly through bash"
exit 1
fi
export PYVERSIONS='3.6.15 pypy3.6-7.3.1 3.7.16 pypy3.7-7.3.9 pypy3.8-7.3.10 pyston-2.3.5 3.8.18'

View File

@@ -1,8 +0,0 @@
# -*- shell-script -*-
# Sets PYVERSIONS to be pyenv versions that
# we can use in the master branch.
if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
echo "This script should be *sourced* rather than run directly through bash"
exit 1
fi
export PYVERSIONS='3.7.13 pyston-2.3.3 3.8.13'

View File

@@ -1,28 +0,0 @@
#!/bin/bash
# Runs test_pyenvlib.test on all versions of Python master.
function finish {
cd $owd
}
# FIXME put some of the below in a common routine
owd=$(pwd)
trap finish EXIT
cd $(dirname ${BASH_SOURCE[0]})
if ! source ./pyenv-newer-versions ; then
exit $?
fi
if ! source ./setup-master.sh ; then
exit $?
fi
cd ../test
for version in $PYVERSIONS; do
if ! pyenv local $version ; then
exit $?
fi
echo "====== Running test_pyenvlib.py on $version ====="
if ! python ./test_pyenvlib.py --weak-verify --max 800 --${version} ; then
exit $?
fi
echo "------ Done test_pyenvlib.py on $version -----"
done

View File

@@ -1,20 +0,0 @@
#!/bin/bash
# Check out master branch and dependent development master branches
bs=${BASH_SOURCE[0]}
if [[ $0 == $bs ]] ; then
echo "This script should be *sourced* rather than run directly through bash"
exit 1
fi
PYTHON_VERSION=3.12
uncompyle6_owd=$(pwd)
mydir=$(dirname $bs)
fulldir=$(readlink -f $mydir)
cd $mydir
. ./checkout_common.sh
cd $fulldir/..
(cd $fulldir/.. && \
setup_version python-spark master && \
setup_version python-xdis master )
checkout_finish master

View File

@@ -1,23 +0,0 @@
#!/bin/bash
# Check out python-2.4-to-2.7 and dependent development branches.
bs=${BASH_SOURCE[0]}
if [[ $0 == $bs ]] ; then
echo "This script should be *sourced* rather than run directly through bash"
exit 1
fi
PYTHON_VERSION=2.4
uncompyle6_owd=$(pwd)
mydir=$(dirname $bs)
fulldir=$(readlink -f $mydir)
cd $mydir
. ./checkout_common.sh
(cd $fulldir/.. && \
setup_version python-spark python-2.4 && \
setup_version python-xdis python-2.4)
checkout_finish python-2.4-to-2.7

View File

@@ -1,20 +0,0 @@
#!/bin/bash
# Check out python-3.0-to-3.2 and dependent development branches.
bs=${BASH_SOURCE[0]}
if [[ $0 == $bs ]] ; then
echo "This script should be *sourced* rather than run directly through bash"
exit 1
fi
PYTHON_VERSION=3.0
uncompyle6_owd=$(pwd)
mydir=$(dirname $bs)
fulldir=$(readlink -f $mydir)
cd $mydir
. ./checkout_common.sh
(cd $fulldir/.. && \
setup_version python-spark python-3.0 && \
setup_version python-xdis python-3.0)
checkout_finish python-3.0-to-3.2

View File

@@ -1,21 +0,0 @@
#!/bin/bash
# Check out python-3.3-to-3.5 and dependent development branches.
bs=${BASH_SOURCE[0]}
if [[ $0 == $bs ]] ; then
echo "This script should be *sourced* rather than run directly through bash"
exit 1
fi
PYTHON_VERSION=3.3
uncompyle6_owd=$(pwd)
mydir=$(dirname $bs)
cd $mydir
fulldir=$(readlink -f $mydir)
. ./checkout_common.sh
cd $fulldir/..
(cd $fulldir/.. && \
setup_version python-spark python-3.3 && \
setup_version python-xdis python-3.3 )
checkout_finish python-3.3-to-3.5

View File

@@ -1,21 +0,0 @@
#!/bin/bash
# Check out python-3.6-to-3.10 and dependent development branches.
bs=${BASH_SOURCE[0]}
if [[ $0 == $bs ]] ; then
echo "This script should be *sourced* rather than run directly through bash"
exit 1
fi
PYTHON_VERSION=3.6
uncompyle6_owd=$(pwd)
mydir=$(dirname $bs)
cd $mydir
fulldir=$(readlink -f $mydir)
. ./checkout_common.sh
cd $fulldir/..
(cd $fulldir/.. && \
setup_version python-spark python-3.6 && \
setup_version python-xdis python-3.6 )
checkout_finish python-3.6-to-3.10

View File

@@ -1,32 +0,0 @@
#!/bin/bash
# Use pycdc to run our test/bytecode_2.7* test suite
bs=${BASH_SOURCE[0]}
topdir=$(dirname $bs)/..
(cd $topdir && pyenv local 2.7.14)
testdir=$topdir/test
fulldir=$(readlink -f $testdir)
cd $fulldir
for bytecode in bytecode_2.7/*.pyc ; do
echo $bytecode
uncompyle2 $bytecode > /dev/null
echo ================ $bytecode rc: $? ==============
done
tmpdir=/tmp/test-2.7
( cd bytecode_2.7_run &&
mkdir $tmpdir || true
for bytecode in *.pyc ; do
shortname=$(basename $bytecode .pyc)
echo $bytecode
py_file=${tmpdir}/${shortname}.py
typeset -i rc=0
uncompyle2 $bytecode > $py_file
rc=$?
if (( rc == 0 )); then
python $py_file
rc=$?
fi
echo ================ $bytecode rc: $rc ==============
done
)

View File

@@ -1,229 +0,0 @@
# Sample script to install Python and pip under Windows
# Authors: Olivier Grisel, Jonathan Helmus, Kyle Kastner, and Alex Willmer
# License: CC0 1.0 Universal: http://creativecommons.org/publicdomain/zero/1.0/
$MINICONDA_URL = "http://repo.continuum.io/miniconda/"
$BASE_URL = "https://www.python.org/ftp/python/"
$GET_PIP_URL = "https://bootstrap.pypa.io/get-pip.py"
$GET_PIP_PATH = "C:\get-pip.py"
$PYTHON_PRERELEASE_REGEX = @"
(?x)
(?<major>\d+)
\.
(?<minor>\d+)
\.
(?<micro>\d+)
(?<prerelease>[a-z]{1,2}\d+)
"@
function Download ($filename, $url) {
$webclient = New-Object System.Net.WebClient
$basedir = $pwd.Path + "\"
$filepath = $basedir + $filename
if (Test-Path $filename) {
Write-Host "Reusing" $filepath
return $filepath
}
# Download and retry up to 3 times in case of network transient errors.
Write-Host "Downloading" $filename "from" $url
$retry_attempts = 2
for ($i = 0; $i -lt $retry_attempts; $i++) {
try {
$webclient.DownloadFile($url, $filepath)
break
}
Catch [Exception]{
Start-Sleep 1
}
}
if (Test-Path $filepath) {
Write-Host "File saved at" $filepath
} else {
# Retry once to get the error message if any at the last try
$webclient.DownloadFile($url, $filepath)
}
return $filepath
}
function ParsePythonVersion ($python_version) {
if ($python_version -match $PYTHON_PRERELEASE_REGEX) {
return ([int]$matches.major, [int]$matches.minor, [int]$matches.micro,
$matches.prerelease)
}
$version_obj = [version]$python_version
return ($version_obj.major, $version_obj.minor, $version_obj.build, "")
}
function DownloadPython ($python_version, $platform_suffix) {
$major, $minor, $micro, $prerelease = ParsePythonVersion $python_version
if (($major -le 2 -and $micro -eq 0) `
-or ($major -eq 3 -and $minor -le 2 -and $micro -eq 0) `
) {
$dir = "$major.$minor"
$python_version = "$major.$minor$prerelease"
} else {
$dir = "$major.$minor.$micro"
}
if ($prerelease) {
if (($major -le 2) `
-or ($major -eq 3 -and $minor -eq 1) `
-or ($major -eq 3 -and $minor -eq 2) `
-or ($major -eq 3 -and $minor -eq 3) `
) {
$dir = "$dir/prev"
}
}
if (($major -le 2) -or ($major -le 3 -and $minor -le 4)) {
$ext = "msi"
if ($platform_suffix) {
$platform_suffix = ".$platform_suffix"
}
} else {
$ext = "exe"
if ($platform_suffix) {
$platform_suffix = "-$platform_suffix"
}
}
$filename = "python-$python_version$platform_suffix.$ext"
$url = "$BASE_URL$dir/$filename"
$filepath = Download $filename $url
return $filepath
}
function InstallPython ($python_version, $architecture, $python_home) {
Write-Host "Installing Python" $python_version "for" $architecture "bit architecture to" $python_home
if (Test-Path $python_home) {
Write-Host $python_home "already exists, skipping."
return $false
}
if ($architecture -eq "32") {
$platform_suffix = ""
} else {
$platform_suffix = "amd64"
}
$installer_path = DownloadPython $python_version $platform_suffix
$installer_ext = [System.IO.Path]::GetExtension($installer_path)
Write-Host "Installing $installer_path to $python_home"
$install_log = $python_home + ".log"
if ($installer_ext -eq '.msi') {
InstallPythonMSI $installer_path $python_home $install_log
} else {
InstallPythonEXE $installer_path $python_home $install_log
}
if (Test-Path $python_home) {
Write-Host "Python $python_version ($architecture) installation complete"
} else {
Write-Host "Failed to install Python in $python_home"
Get-Content -Path $install_log
Exit 1
}
}
function InstallPythonEXE ($exepath, $python_home, $install_log) {
$install_args = "/quiet InstallAllUsers=1 TargetDir=$python_home"
RunCommand $exepath $install_args
}
function InstallPythonMSI ($msipath, $python_home, $install_log) {
$install_args = "/qn /log $install_log /i $msipath TARGETDIR=$python_home"
$uninstall_args = "/qn /x $msipath"
RunCommand "msiexec.exe" $install_args
if (-not(Test-Path $python_home)) {
Write-Host "Python seems to be installed else-where, reinstalling."
RunCommand "msiexec.exe" $uninstall_args
RunCommand "msiexec.exe" $install_args
}
}
function RunCommand ($command, $command_args) {
Write-Host $command $command_args
Start-Process -FilePath $command -ArgumentList $command_args -Wait -Passthru
}
function InstallPip ($python_home) {
$pip_path = $python_home + "\Scripts\pip.exe"
$python_path = $python_home + "\python.exe"
if (-not(Test-Path $pip_path)) {
Write-Host "Installing pip..."
$webclient = New-Object System.Net.WebClient
$webclient.DownloadFile($GET_PIP_URL, $GET_PIP_PATH)
Write-Host "Executing:" $python_path $GET_PIP_PATH
& $python_path $GET_PIP_PATH
} else {
Write-Host "pip already installed."
}
}
function DownloadMiniconda ($python_version, $platform_suffix) {
if ($python_version -eq "3.4") {
$filename = "Miniconda3-3.5.5-Windows-" + $platform_suffix + ".exe"
} else {
$filename = "Miniconda-3.5.5-Windows-" + $platform_suffix + ".exe"
}
$url = $MINICONDA_URL + $filename
$filepath = Download $filename $url
return $filepath
}
function InstallMiniconda ($python_version, $architecture, $python_home) {
Write-Host "Installing Python" $python_version "for" $architecture "bit architecture to" $python_home
if (Test-Path $python_home) {
Write-Host $python_home "already exists, skipping."
return $false
}
if ($architecture -eq "32") {
$platform_suffix = "x86"
} else {
$platform_suffix = "x86_64"
}
$filepath = DownloadMiniconda $python_version $platform_suffix
Write-Host "Installing" $filepath "to" $python_home
$install_log = $python_home + ".log"
$args = "/S /D=$python_home"
Write-Host $filepath $args
Start-Process -FilePath $filepath -ArgumentList $args -Wait -Passthru
if (Test-Path $python_home) {
Write-Host "Python $python_version ($architecture) installation complete"
} else {
Write-Host "Failed to install Python in $python_home"
Get-Content -Path $install_log
Exit 1
}
}
function InstallMinicondaPip ($python_home) {
$pip_path = $python_home + "\Scripts\pip.exe"
$conda_path = $python_home + "\Scripts\conda.exe"
if (-not(Test-Path $pip_path)) {
Write-Host "Installing pip..."
$args = "install --yes pip"
Write-Host $conda_path $args
Start-Process -FilePath "$conda_path" -ArgumentList $args -Wait -Passthru
} else {
Write-Host "pip already installed."
}
}
function main () {
InstallPython $env:PYTHON_VERSION $env:PYTHON_ARCH $env:PYTHON
InstallPip $env:PYTHON
}
main

View File

@@ -1,87 +0,0 @@
:: To build extensions for 64 bit Python 3, we need to configure environment
:: variables to use the MSVC 2010 C++ compilers from GRMSDKX_EN_DVD.iso of:
:: MS Windows SDK for Windows 7 and .NET Framework 4 (SDK v7.1)
::
:: To build extensions for 64 bit Python 2, we need to configure environment
:: variables to use the MSVC 2008 C++ compilers from GRMSDKX_EN_DVD.iso of:
:: MS Windows SDK for Windows 7 and .NET Framework 3.5 (SDK v7.0)
::
:: 32 bit builds, and 64-bit builds for 3.5 and beyond, do not require specific
:: environment configurations.
::
:: Note: this script needs to be run with the /E:ON and /V:ON flags for the
:: cmd interpreter, at least for (SDK v7.0)
::
:: More details at:
:: https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows
:: http://stackoverflow.com/a/13751649/163740
::
:: Author: Olivier Grisel
:: License: CC0 1.0 Universal: http://creativecommons.org/publicdomain/zero/1.0/
::
:: Notes about batch files for Python people:
::
:: Quotes in values are literally part of the values:
:: SET FOO="bar"
:: FOO is now five characters long: " b a r "
:: If you don't want quotes, don't include them on the right-hand side.
::
:: The CALL lines at the end of this file look redundant, but if you move them
:: outside of the IF clauses, they do not run properly in the SET_SDK_64==Y
:: case, I don't know why.
@ECHO OFF
SET COMMAND_TO_RUN=%*
SET WIN_SDK_ROOT=C:\Program Files\Microsoft SDKs\Windows
SET WIN_WDK=c:\Program Files (x86)\Windows Kits\10\Include\wdf
:: Extract the major and minor versions, and allow for the minor version to be
:: more than 9. This requires the version number to have two dots in it.
SET MAJOR_PYTHON_VERSION=%PYTHON_VERSION:~0,1%
IF "%PYTHON_VERSION:~3,1%" == "." (
SET MINOR_PYTHON_VERSION=%PYTHON_VERSION:~2,1%
) ELSE (
SET MINOR_PYTHON_VERSION=%PYTHON_VERSION:~2,2%
)
:: Based on the Python version, determine what SDK version to use, and whether
:: to set the SDK for 64-bit.
IF %MAJOR_PYTHON_VERSION% == 2 (
SET WINDOWS_SDK_VERSION="v7.0"
SET SET_SDK_64=Y
) ELSE (
IF %MAJOR_PYTHON_VERSION% == 3 (
SET WINDOWS_SDK_VERSION="v7.1"
IF %MINOR_PYTHON_VERSION% LEQ 4 (
SET SET_SDK_64=Y
) ELSE (
SET SET_SDK_64=N
IF EXIST "%WIN_WDK%" (
:: See: https://connect.microsoft.com/VisualStudio/feedback/details/1610302/
REN "%WIN_WDK%" 0wdf
)
)
) ELSE (
ECHO Unsupported Python version: "%MAJOR_PYTHON_VERSION%"
EXIT 1
)
)
IF %PYTHON_ARCH% == 64 (
IF %SET_SDK_64% == Y (
ECHO Configuring Windows SDK %WINDOWS_SDK_VERSION% for Python %MAJOR_PYTHON_VERSION% on a 64 bit architecture
SET DISTUTILS_USE_SDK=1
SET MSSdk=1
"%WIN_SDK_ROOT%\%WINDOWS_SDK_VERSION%\Setup\WindowsSdkVer.exe" -q -version:%WINDOWS_SDK_VERSION%
"%WIN_SDK_ROOT%\%WINDOWS_SDK_VERSION%\Bin\SetEnv.cmd" /x64 /release
ECHO Executing: %COMMAND_TO_RUN%
call %COMMAND_TO_RUN% || EXIT 1
) ELSE (
ECHO Using default MSVC build environment for 64 bit architecture
ECHO Executing: %COMMAND_TO_RUN%
call %COMMAND_TO_RUN% || EXIT 1
)
) ELSE (
ECHO Using default MSVC build environment for 32 bit architecture
ECHO Executing: %COMMAND_TO_RUN%
call %COMMAND_TO_RUN% || EXIT 1
)

13
circle.yml Normal file
View File

@@ -0,0 +1,13 @@
machine:
python:
version: 2.7.10
environment:
COMPILE: --compile
dependencies:
override:
- pip install -r requirements.txt
- pip install -r requirements-dev.txt
test:
override:
- python ./setup.py develop && make check-2.6

1
pytest/.gitignore vendored
View File

@@ -1,2 +1 @@
/.hypothesis
/__pycache__

View File

@@ -1,11 +1,7 @@
PHONY=check test pytest
SHELL=/bin/bash
PYTHON ?= python
#: Run all tests
test check pytest:
@PYTHON_VERSION=`$(PYTHON) -V 2>&1 | cut -d ' ' -f 2 | cut -d'.' -f1,2`; \
if [[ $$PYTHON_VERSION > 3.2 ]] || [[ $$PYTHON_VERSION == 2.7 ]] || [[ $$PYTHON_VERSION == 2.6 ]]; then \
$(PYTHON) -m pytest .; \
fi
py.test

View File

@@ -1,6 +0,0 @@
source ../.venv.3.6/bin/activate
py.test -k test_CALL_FUNCTION_KW
source ../.venv.3.5/bin/activate
py.test -k test_CALL_FUNCTION_KW
source ../.venv.2.7/bin/activate
py.test -k test_CALL_FUNCTION_KW

View File

@@ -1,11 +0,0 @@
from uncompyle6.scanner import get_scanner
from uncompyle6.parser import get_python_parser
def test_get_scanner():
# See that we can retrieve a scanner using a full version number
assert get_scanner('2.7.13')
def test_get_parser():
# See that we can retrieve a sparser using a full version number
assert get_python_parser('2.7.13')

View File

@@ -1,21 +0,0 @@
import pytest
# uncompyle6
from xdis.version_info import PYTHON_VERSION_TRIPLE, IS_PYPY
from validate import validate_uncompyle
@pytest.mark.skipif(PYTHON_VERSION_TRIPLE < (3, 6) or IS_PYPY, reason="need at least Python 3.6 and not PyPY")
@pytest.mark.parametrize('text', (
"{0.: 'a', -1: 'b'}", # BUILD_MAP
"{'a':'b'}", # BUILD_MAP
"{0: 1}", # BUILD_MAP
"{b'0':1, b'2':3}", # BUILD_CONST_KEY_MAP
"{0: 1, 2: 3}", # BUILD_CONST_KEY_MAP
"{'a':'b','c':'d'}", # BUILD_CONST_KEY_MAP
"{0: 1, 2: 3}", # BUILD_CONST_KEY_MAP
"{'a': 1, 'b': 2}", # BUILD_CONST_KEY_MAP
"{'a':'b','c':'d'}", # BUILD_CONST_KEY_MAP
"{0.0:'b',0.1:'d'}", # BUILD_CONST_KEY_MAP
))
def test_build_const_key_map(text):
validate_uncompyle(text)

View File

@@ -1,5 +1,5 @@
from uncompyle6.semantics.fragments import code_deparse as deparse
from xdis.version_info import PYTHON_VERSION_TRIPLE
from uncompyle6.semantics.fragments import deparse_code as deparse
from uncompyle6 import PYTHON_VERSION, PYTHON3
def map_stmts(x, y):
x = []
@@ -29,23 +29,22 @@ def list_comp():
[y for y in range(3)]
def get_parsed_for_fn(fn):
code = fn.__code__
return deparse(code, version=PYTHON_VERSION_TRIPLE)
code = fn.__code__ if PYTHON3 else fn.func_code
return deparse(PYTHON_VERSION, code)
def check_expect(expect, parsed, fn_name):
def check_expect(expect, parsed):
debug = False
i = 2
max_expect = len(expect)
for name, offset in sorted(parsed.offsets.keys()):
assert i+1 <= max_expect, (
"%s: ran out if items in testing node" % fn_name)
assert i+1 <= max_expect, "ran out if items in testing node"
nodeInfo = parsed.offsets[name, offset]
node = nodeInfo.node
extractInfo = parsed.extract_node_info(node)
assert expect[i] == extractInfo.selectedLine, \
('%s: line %s expect:\n%s\ngot:\n%s' %
(fn_name, i, expect[i], extractInfo.selectedLine))
('line %s expect:\n%s\ngot:\n%s' %
(i, expect[i], extractInfo.selectedLine))
assert expect[i+1] == extractInfo.markerLine, \
('line %s expect:\n%s\ngot:\n%s' %
(i+1, expect[i+1], extractInfo.markerLine))
@@ -74,7 +73,6 @@ def check_expect(expect, parsed, fn_name):
def test_stuff():
return
parsed = get_parsed_for_fn(map_stmts)
expect = """
-1
@@ -85,10 +83,10 @@ return (x, y)
-------------
0
x = []
-
--
Contained in...
x = []
--
------
3
x = []
-
@@ -97,10 +95,10 @@ x = []
------
6
y = {}
-
--
Contained in...
y = {}
--
------
9
y = {}
-
@@ -132,7 +130,7 @@ Contained in...
x = [] ...
------ ...
""".split("\n")
check_expect(expect, parsed, 'map_stmts')
check_expect(expect, parsed)
########################################################
# return
@@ -169,7 +167,7 @@ Contained in...
return (x, y)
-------------
""".split("\n")
check_expect(expect, parsed, 'return_stmt')
check_expect(expect, parsed)
########################################################
# # try
@@ -316,3 +314,5 @@ for i in range(2): ...
.
""".split("\n")
parsed = get_parsed_for_fn(for_range_stmt)
if not PYTHON3:
check_expect(expect, parsed)

View File

@@ -1,27 +0,0 @@
import os.path
import pytest
from uncompyle6.code_fns import disassemble_file
def get_srcdir():
filename = os.path.normcase(os.path.dirname(__file__))
return os.path.realpath(filename)
src_dir = get_srcdir()
os.chdir(src_dir)
@pytest.mark.parametrize(("test_tuple"), [
('../test/bytecode_2.7/05_if.pyc', 'testdata/if-2.7.right',),
('../test/bytecode_2.7/05_ifelse.pyc', 'testdata/ifelse-2.7.right',),
])
def test_funcoutput(capfd, test_tuple):
in_file, filename_expected = test_tuple
disassemble_file(in_file)
resout, reserr = capfd.readouterr()
expected = open(filename_expected, "r").read()
if resout != expected:
with open(filename_expected + ".got", "w") as out:
out.write(resout)
assert resout == expected

View File

@@ -0,0 +1,33 @@
import os.path
import pytest
from uncompyle6.disas import disassemble_file
def get_srcdir():
filename = os.path.normcase(os.path.dirname(__file__))
return os.path.realpath(filename)
src_dir = get_srcdir()
os.chdir(src_dir)
@pytest.mark.parametrize(("test_tuple", "function_to_test"), [
(
('../test/bytecode_2.7/05_if.pyc', 'testdata/if-2.7.right',),
disassemble_file
),
(
('../test/bytecode_2.7/05_ifelse.pyc', 'testdata/ifelse-2.7.right',),
disassemble_file
),
])
def test_funcoutput(capfd, test_tuple, function_to_test):
in_file , filename_expected = test_tuple
function_to_test(in_file, native=False)
resout, reserr = capfd.readouterr()
expected = open(filename_expected, "r").read()
if resout != expected:
with open(filename_expected + ".got", "w") as out:
out.write(resout)
assert resout == expected

78
pytest/test_docstring.py Normal file
View File

@@ -0,0 +1,78 @@
import sys
from uncompyle6 import PYTHON3
if PYTHON3:
from io import StringIO
minint = -sys.maxsize-1
maxint = sys.maxsize
else:
from StringIO import StringIO
minint = -sys.maxint-1
maxint = sys.maxint
from uncompyle6.semantics.helper import print_docstring
class PrintFake():
def __init__(self):
self.pending_newlines = 0
self.f = StringIO()
def write(self, *data):
if (len(data) == 0) or (len(data) == 1 and data[0] == ''):
return
out = ''.join((str(j) for j in data))
n = 0
for i in out:
if i == '\n':
n += 1
if n == len(out):
self.pending_newlines = max(self.pending_newlines, n)
return
elif n:
self.pending_newlines = max(self.pending_newlines, n)
out = out[n:]
break
else:
break
if self.pending_newlines > 0:
self.f.write('\n'*self.pending_newlines)
self.pending_newlines = 0
for i in out[::-1]:
if i == '\n':
self.pending_newlines += 1
else:
break
if self.pending_newlines:
out = out[:-self.pending_newlines]
self.f.write(out)
def println(self, *data):
if data and not(len(data) == 1 and data[0] ==''):
self.write(*data)
self.pending_newlines = max(self.pending_newlines, 1)
return
pass
def test_docstring():
for doc, expect in (
("Now is the time",
' """Now is the time"""'),
("""
Now is the time
""",
''' """
Now is the time
"""''')
# (r'''func placeholder - ' and with ("""\nstring\n """)''',
# """ r'''func placeholder - ' and with (\"\"\"\nstring\n\"\"\")'''"""),
# (r"""func placeholder - ' and with ('''\nstring\n''') and \"\"\"\nstring\n\"\"\" """,
# """ r\"\"\"func placeholder - ' and with ('''\nstring\n''') and \"\"\"\nstring\n\"\"\" \"\"\"""")
):
o = PrintFake()
# print(doc)
# print(expect)
print_docstring(o, ' ', doc)
assert expect == o.f.getvalue()

View File

@@ -1,6 +1,7 @@
#!/usr/bin/env python
from xdis.version_info import PYTHON_VERSION_TRIPLE, IS_PYPY, version_tuple_to_str
from uncompyle6 import PYTHON_VERSION, IS_PYPY
from uncompyle6.scanner import get_scanner
from array import array
def bug(state, slotstate):
if state:
if slotstate is not None:
@@ -21,24 +22,24 @@ def bug_loop(disassemble, tb=None):
def test_if_in_for():
code = bug.__code__
scan = get_scanner(PYTHON_VERSION_TRIPLE)
if (2, 7) <= PYTHON_VERSION_TRIPLE < (3, 1) and not IS_PYPY:
scan.build_instructions(code)
scan = get_scanner(PYTHON_VERSION)
print(PYTHON_VERSION)
if 2.7 <= PYTHON_VERSION <= 3.0 and not IS_PYPY:
n = scan.setup_code(code)
scan.build_lines_data(code, n)
scan.build_prev_op(n)
fjt = scan.find_jump_targets(False)
## FIXME: the data below is wrong.
## we get different results currently as well.
## We need to probably fix both the code
## and the test below
# assert {15: [3], 69: [66], 63: [18]} == fjt
# assert scan.structs == \
# [{'start': 0, 'end': 72, 'type': 'root'},
# {'start': 15, 'end': 66, 'type': 'if-then'},
# {'start': 31, 'end': 59, 'type': 'for-loop'},
# {'start': 62, 'end': 63, 'type': 'for-else'}]
assert {15: [3], 69: [66], 63: [18]} == fjt
assert scan.structs == \
[{'start': 0, 'end': 72, 'type': 'root'},
{'start': 15, 'end': 66, 'type': 'if-then'},
{'start': 31, 'end': 59, 'type': 'for-loop'},
{'start': 62, 'end': 63, 'type': 'for-else'}]
code = bug_loop.__code__
scan.build_instructions(code)
n = scan.setup_code(code)
scan.build_lines_data(code, n)
scan.build_prev_op(n)
fjt = scan.find_jump_targets(False)
assert{64: [42], 67: [42, 42], 42: [16, 41], 19: [6]} == fjt
assert scan.structs == [
@@ -51,8 +52,10 @@ def test_if_in_for():
# previous bug was not mistaking while-loop for if-then
{'start': 48, 'end': 67, 'type': 'while-loop'}]
elif (3, 2) < PYTHON_VERSION_TRIPLE <= (3, 4):
scan.build_instructions(code)
elif 3.2 < PYTHON_VERSION <= 3.4:
scan.code = array('B', code.co_code)
scan.build_lines_data(code)
scan.build_prev_op()
fjt = scan.find_jump_targets(False)
assert {69: [66], 63: [18]} == fjt
assert scan.structs == \
@@ -62,6 +65,5 @@ def test_if_in_for():
{'end': 59, 'type': 'for-loop', 'start': 31},
{'end': 63, 'type': 'for-else', 'start': 62}]
else:
print("FIXME: should fix for %s" % version_tuple_to_str())
assert True
assert True, "FIXME: should note fixed"
return

View File

@@ -1,148 +1,63 @@
import re
from uncompyle6 import PYTHON_VERSION, PYTHON3, IS_PYPY # , PYTHON_VERSION
from uncompyle6.parser import get_python_parser, python_parser
from uncompyle6.scanner import get_scanner
from xdis.version_info import PYTHON_VERSION_TRIPLE, IS_PYPY
def test_grammar():
def check_tokens(tokens, opcode_set):
remain_tokens = set(tokens) - opcode_set
remain_tokens = set([re.sub(r"_\d+$", "", t) for t in remain_tokens])
remain_tokens = set([re.sub("_CONT$", "", t) for t in remain_tokens])
remain_tokens = set([re.sub("LOAD_CODE$", "", t) for t in remain_tokens])
remain_tokens = set([re.sub('_\d+$','', t) for t in remain_tokens])
remain_tokens = set([re.sub('_CONT$','', t) for t in remain_tokens])
remain_tokens = set(remain_tokens) - opcode_set
assert remain_tokens == set([]), "Remaining tokens %s\n====\n%s" % (
remain_tokens,
p.dump_grammar(),
)
assert remain_tokens == set([]), \
"Remaining tokens %s\n====\n%s" % (remain_tokens, p.dumpGrammar())
p = get_python_parser(PYTHON_VERSION_TRIPLE, is_pypy=IS_PYPY)
(lhs, rhs, tokens, right_recursive, dup_rhs) = p.check_sets()
p = get_python_parser(PYTHON_VERSION, is_pypy=IS_PYPY)
lhs, rhs, tokens, right_recursive = p.checkSets()
expect_lhs = set(['expr1024', 'pos_arg'])
unused_rhs = set(['build_list', 'call_function', 'mkfunc',
'mklambda',
'unpack', 'unpack_list'])
expect_right_recursive = [['designList', ('designator', 'DUP_TOP', 'designList')]]
if PYTHON3:
expect_lhs.add('load_genexpr')
# We have custom rules that create the below
expect_lhs = set(["pos_arg"])
if PYTHON_VERSION_TRIPLE < (3, 8):
if PYTHON_VERSION_TRIPLE < (3, 7):
expect_lhs.add("attribute")
expect_lhs.add("get_iter")
if PYTHON_VERSION_TRIPLE >= (3, 8) or PYTHON_VERSION_TRIPLE < (3, 0):
expect_lhs.add("stmts_opt")
unused_rhs = unused_rhs.union(set("""
except_pop_except genexpr classdefdeco2 listcomp
""".split()))
if 3.0 <= PYTHON_VERSION:
expect_lhs.add("annotate_arg")
expect_lhs.add("annotate_tuple")
unused_rhs.add("mkfunc_annotate")
pass
else:
expect_lhs.add("async_with_as_stmt")
expect_lhs.add("async_with_stmt")
unused_rhs = set(["list", "mkfunc", "lambda_body", "unpack"])
expect_right_recursive = set([("designList", ("store", "DUP_TOP", "designList"))])
if PYTHON_VERSION_TRIPLE[:2] <= (3, 6):
unused_rhs.add("call")
if PYTHON_VERSION_TRIPLE >= (2, 7):
expect_lhs.add("kvlist")
expect_lhs.add("kv3")
unused_rhs.add("dict")
if PYTHON_VERSION_TRIPLE < (3, 7) and PYTHON_VERSION_TRIPLE[:2] != (2, 7):
# NOTE: this may disappear
expect_lhs.add("except_handler_else")
expect_lhs.add("load_genexpr")
unused_rhs = unused_rhs.union(
set(
"""
except_pop_except generator_exp
""".split()
)
)
if PYTHON_VERSION_TRIPLE < (3, 7):
expect_lhs.add("annotate_arg")
expect_lhs.add("annotate_tuple")
unused_rhs.add("mkfunc_annotate")
unused_rhs.add("dict_comp")
unused_rhs.add("classdefdeco1")
unused_rhs.add("tryelsestmtl")
if PYTHON_VERSION_TRIPLE >= (3, 5):
expect_right_recursive.add(
(("l_stmts", ("lastl_stmt", "come_froms", "l_stmts")))
)
pass
pass
if PYTHON_VERSION_TRIPLE >= (3, 7):
expect_lhs.add("set_for")
unused_rhs.add("set_iter")
pass
pass
# FIXME
if PYTHON_VERSION_TRIPLE < (3, 8):
assert expect_lhs == set(lhs)
assert unused_rhs == set(rhs)
expect_lhs.add('kwarg')
assert expect_lhs == set(lhs)
assert unused_rhs == set(rhs)
assert expect_right_recursive == right_recursive
expect_dup_rhs = frozenset(
[
("COME_FROM",),
("CONTINUE",),
("JUMP_ABSOLUTE",),
("LOAD_CONST",),
("JUMP_BACK",),
("JUMP_FORWARD",),
]
)
reduced_dup_rhs = dict((k, dup_rhs[k]) for k in dup_rhs if k not in expect_dup_rhs)
if reduced_dup_rhs:
print("\nPossible duplicate RHS that might be folded, into one of the LHS symbols")
for k in reduced_dup_rhs:
print(k, reduced_dup_rhs[k])
# assert not reduced_dup_rhs, reduced_dup_rhs
s = get_scanner(PYTHON_VERSION_TRIPLE, IS_PYPY)
s = get_scanner(PYTHON_VERSION, IS_PYPY)
ignore_set = set(
"""
JUMP_BACK CONTINUE
COME_FROM COME_FROM_EXCEPT
COME_FROM_EXCEPT_CLAUSE
COME_FROM_LOOP COME_FROM_WITH
COME_FROM_FINALLY ELSE
LOAD_GENEXPR LOAD_ASSERT LOAD_SETCOMP LOAD_DICTCOMP LOAD_STR LOAD_CODE
LAMBDA_MARKER
RETURN_END_IF RETURN_END_IF_LAMBDA RETURN_VALUE_LAMBDA RETURN_LAST
""".split()
)
if (2, 6) <= PYTHON_VERSION_TRIPLE <= (2, 7):
opcode_set = set(s.opc.opname).union(ignore_set)
if PYTHON_VERSION_TRIPLE[:2] == (2, 6):
opcode_set.add("THEN")
check_tokens(tokens, opcode_set)
elif PYTHON_VERSION_TRIPLE[:2] == (3, 4):
ignore_set.add("LOAD_ARG") # Used in grammar for comprehension. But not in 3.4
ignore_set.add("LOAD_CLASSNAME")
ignore_set.add("STORE_LOCALS")
"""
JUMP_BACK CONTINUE RETURN_END_IF
COME_FROM COME_FROM_EXCEPT COME_FROM_LOOP COME_FROM_WITH
COME_FROM_FINALLY
LOAD_GENEXPR LOAD_ASSERT LOAD_SETCOMP LOAD_DICTCOMP
LAMBDA_MARKER RETURN_LAST
""".split())
if 2.6 <= PYTHON_VERSION <= 2.7:
opcode_set = set(s.opc.opname).union(ignore_set)
check_tokens(tokens, opcode_set)
elif PYTHON_VERSION == 3.4:
ignore_set.add('LOAD_CLASSNAME')
ignore_set.add('STORE_LOCALS')
opcode_set = set(s.opc.opname).union(ignore_set)
check_tokens(tokens, opcode_set)
def test_dup_rule():
import inspect
python_parser(
PYTHON_VERSION_TRIPLE,
inspect.currentframe().f_code,
is_pypy=IS_PYPY,
parser_debug={
"dups": True,
"transition": False,
"reduce": False,
"rules": False,
"errorstack": None,
"context": True,
},
)
python_parser(PYTHON_VERSION, inspect.currentframe().f_code,
is_pypy=IS_PYPY,
parser_debug={
'dups': True, 'transition': False, 'reduce': False,
'rules': False, 'errorstack': None, 'context': True})

View File

@@ -1,189 +0,0 @@
import sys
from uncompyle6.scanner import get_scanner
from uncompyle6.semantics.consts import (
escape, NONE,
# RETURN_NONE, PASS, RETURN_LOCALS
)
from io import StringIO
def iteritems(d):
return d.items()
from uncompyle6.semantics.pysource import (SourceWalker, deparse_code2str)
def test_template_engine():
s = StringIO()
sys_version = float(sys.version[0:3])
scanner = get_scanner(sys_version, is_pypy=False)
scanner.insts = []
sw = SourceWalker(2.7, s, scanner)
sw.ast = NONE
sw.template_engine(('--%c--', 0), NONE)
print(sw.f.getvalue())
assert sw.f.getvalue() == '--None--'
# FIXME: and so on...
from uncompyle6.semantics.consts import (
TABLE_DIRECT, TABLE_R,
)
from uncompyle6.semantics.fragments import (
TABLE_DIRECT_FRAGMENT,
)
skip_for_now = "DELETE_DEREF".split()
def test_tables():
for t, name, fragment in (
(TABLE_DIRECT, 'TABLE_DIRECT', False),
(TABLE_R, 'TABLE_R', False),
(TABLE_DIRECT_FRAGMENT, 'TABLE_DIRECT_FRAGMENT', True)):
for k, entry in iteritems(t):
if k in skip_for_now:
continue
fmt = entry[0]
arg = 1
i = 0
m = escape.search(fmt)
print("%s[%s]" % (name, k))
while m:
i = m.end()
typ = m.group('type') or '{'
if typ in frozenset(['%', '+', '-', '|', ',', '{']):
# No args
pass
elif typ in frozenset(['c', 'p', 'P', 'C', 'D']):
# One arg - should be int or tuple of int
if typ == 'c':
item = entry[arg]
if isinstance(item, tuple):
assert isinstance(item[1], str), (
"%s[%s][%d] kind %s is '%s' should be str but is %s. "
"Full entry: %s" %
(name, k, arg, typ, item[1], type(item[1]), entry)
)
item = item[0]
assert isinstance(item, int), (
"%s[%s][%d] kind %s is '%s' should be an int but is %s. "
"Full entry: %s" %
(name, k, arg, typ, item, type(item), entry)
)
elif typ in frozenset(['C', 'D']):
tup = entry[arg]
assert isinstance(tup, tuple), (
"%s[%s][%d] type %s is %s should be an tuple but is %s. "
"Full entry: %s" %
(name, k, arg, typ, entry[arg], type(entry[arg]), entry)
)
assert len(tup) == 3
for j, x in enumerate(tup[:-1]):
assert isinstance(x, int), (
"%s[%s][%d][%d] type %s is %s should be an tuple but is %s. "
"Full entry: %s" %
(name, k, arg, j, typ, x, type(x), entry)
)
assert isinstance(tup[-1], str) or tup[-1] is None, (
"%s[%s][%d][%d] sep type %s is %s should be an string but is %s. "
"Full entry: %s" %
(name, k, arg, j, typ, tup[-1], type(x), entry)
)
elif typ == 'P':
tup = entry[arg]
assert isinstance(tup, tuple), (
"%s[%s][%d] type %s is %s should be an tuple but is %s. "
"Full entry: %s" %
(name, k, arg, typ, entry[arg], type(entry[arg]), entry)
)
assert len(tup) == 4
for j, x in enumerate(tup[:-2]):
assert isinstance(x, int), (
"%s[%s][%d][%d] type %s is '%s' should be an tuple but is %s. "
"Full entry: %s" %
(name, k, arg, j, typ, x, type(x), entry)
)
assert isinstance(tup[-2], str), (
"%s[%s][%d][%d] sep type %s is '%s' should be an string but is %s. "
"Full entry: %s" %
(name, k, arg, j, typ, x, type(x), entry)
)
assert isinstance(tup[1], int), (
"%s[%s][%d][%d] prec type %s is '%s' should be an int but is %s. "
"Full entry: %s" %
(name, k, arg, j, typ, x, type(x), entry)
)
else:
# Should be a tuple which contains only ints
tup = entry[arg]
assert isinstance(tup, tuple), (
"%s[%s][%d] type %s is '%s' should be an tuple but is %s. "
"Full entry: %s" %
(name, k, arg, typ, entry[arg], type(entry[arg]), entry)
)
assert 2 <= len(tup) <= 3
for j, x in enumerate(tup):
if len(tup) == 3 and j == 1:
assert isinstance(x, str), (
"%s[%s][%d][%d] type '%s' is '%s should be an string but is %s. Full entry: %s" %
(name, k, arg, j, typ, x, type(x), entry)
)
else:
assert isinstance(x, int), (
"%s[%s][%d][%d] type '%s' is '%s should be an int but is %s. Full entry: %s" %
(name, k, arg, j, typ, x, type(x), entry)
)
pass
arg += 1
elif typ in frozenset(['r']) and fragment:
pass
elif typ == 'b' and fragment:
assert isinstance(entry[arg], int), (
"%s[%s][%d] type %s is '%s' should be an int but is %s. "
"Full entry: %s" %
(name, k, arg, typ, entry[arg], type(entry[arg]), entry)
)
arg += 1
elif typ == 'x' and fragment:
tup = entry[arg]
assert isinstance(tup, tuple), (
"%s[%s][%d] type %s is '%s' should be an tuple but is %s. "
"Full entry: %s" %
(name, k, arg, typ, entry[arg], type(entry[arg]), entry)
)
assert len(tup) == 2
assert isinstance(tup[0], int), (
"%s[%s][%d] source type %s is '%s' should be an int but is %s. "
"Full entry: %s" %
(name, k, arg, typ, entry[arg], type(entry[arg]), entry)
)
assert isinstance(tup[1], tuple), (
"%s[%s][%d] dest type %s is '%s' should be an tuple but is %s. "
"Full entry: %s" %
(name, k, arg, typ, entry[arg], type(entry[arg]), entry)
)
for j, x in enumerate(tup[1]):
assert isinstance(x, int), (
"%s[%s][%d][%d] type %s is %s should be an int but is %s. Full entry: %s" %
(name, k, arg, j, typ, x, type(x), entry)
)
arg += 1
pass
else:
assert False, (
"%s[%s][%d] type %s is not known. Full entry: %s" %
(name, k, arg, typ, entry)
)
m = escape.search(fmt, i)
pass
assert arg == len(entry), (
"%s[%s] arg %d should be length of entry %d. Full entry: %s" %
(name, k, arg, len(entry), entry))
def test_deparse_code2str():
def deparse_test(co):
"This is a docstring"
s = deparse_code2str(co, debug_opts={"asm": "after", "tree": True})
assert s
return
deparse_test(deparse_test.__code__)

View File

@@ -1,24 +1,19 @@
import pytest
from uncompyle6 import code_deparse
from xdis.version_info import PYTHON_VERSION_TRIPLE
pytest.mark.skip(PYTHON_VERSION_TRIPLE < (2, 7), reason="need Python < 2.7")
from uncompyle6 import PYTHON_VERSION, PYTHON3, deparse_code
def test_single_mode():
single_expressions = (
"i = 1",
"i and (j or k)",
"i += 1",
"i = j % 4",
"i = {}",
"i = []",
"for i in range(10):\n i\n",
"for i in range(10):\n for j in range(10):\n i + j\n",
# 'try:\n i\nexcept Exception:\n j\nelse:\n k\n'
'i = 1',
'i and (j or k)',
'i += 1',
'i = j % 4',
'i = {}',
'i = []',
'for i in range(10):\n i\n',
'for i in range(10):\n for j in range(10):\n i + j\n',
'try:\n i\nexcept Exception:\n j\nelse:\n k\n'
)
for expr in single_expressions:
code = compile(expr + "\n", "<string>", "single")
got = code_deparse(code, compile_mode="single").text
assert got == expr + "\n"
code = compile(expr + '\n', '<string>', 'single')
assert deparse_code(PYTHON_VERSION, code, compile_mode='single').text == expr + '\n'

View File

@@ -1,24 +0,0 @@
from uncompyle6.scanners.tok import Token
def test_token():
# Test token formatting of: LOAD_CONST None
t = Token("LOAD_CONST", offset=0, attr=None, pattr=None, has_arg=True)
expect = "0 LOAD_CONST None"
# print(t.format())
assert t
assert t.format().strip() == expect.strip()
# Make sure equality testing of tokens ignores offset
t2 = Token("LOAD_CONST", offset=2, attr=None, pattr=None, has_arg=True)
assert t2 == t
# Make sure formatting of: LOAD_CONST False. We assume False is the 0th index
# of co_consts.
t = Token("LOAD_CONST", offset=1, attr=False, pattr=False, has_arg=True)
expect = "1 LOAD_CONST False"
assert t.format().strip() == expect.strip()
if __name__ == "__main__":
test_token()

View File

@@ -1,12 +1,12 @@
# Python 2.7
# Embedded file name: simple_source/branching/05_if.py
L. 6 0 LOAD_NAME 0 'True'
3 POP_JUMP_IF_FALSE 15 'to 15'
6 0 LOAD_NAME 0 'True'
3 POP_JUMP_IF_FALSE 15 'to 15'
L. 7 6 LOAD_NAME 1 'False'
9 STORE_NAME 2 'b'
12 JUMP_FORWARD 0 'to 15'
15_0 COME_FROM 12 '12'
15 LOAD_CONST None
18 RETURN_VALUE
7 6 LOAD_NAME 1 'False'
9 STORE_NAME 2 'b'
12 JUMP_FORWARD 0 'to 15'
15_0 COME_FROM '12'
15 LOAD_CONST 0 ''
18 RETURN_VALUE

View File

@@ -1,15 +1,15 @@
# Python 2.7
# Embedded file name: simple_source/branching/05_ifelse.py
L. 3 0 LOAD_NAME 0 'True'
3 POP_JUMP_IF_FALSE 15 'to 15'
3 0 LOAD_NAME 0 'True'
3 POP_JUMP_IF_FALSE 15 'to 15'
L. 4 6 LOAD_CONST 1
9 STORE_NAME 1 'b'
12 JUMP_FORWARD 6 'to 21'
4 6 LOAD_CONST 0 1
9 STORE_NAME 1 'b'
12 JUMP_FORWARD 6 'to 21'
L. 6 15 LOAD_CONST 2
18 STORE_NAME 2 'd'
21_0 COME_FROM 12 '12'
21 LOAD_CONST None
24 RETURN_VALUE
6 15 LOAD_CONST 1 2
18 STORE_NAME 2 'd'
21_0 COME_FROM '12'
21 LOAD_CONST 2 ''
24 RETURN_VALUE

View File

@@ -1,149 +0,0 @@
# future
from __future__ import print_function
# std
import os
import difflib
import subprocess
import tempfile
import functools
# uncompyle6 / xdis
from uncompyle6 import code_deparse
from xdis.version_info import PYTHON_VERSION_TRIPLE, IS_PYPY
# TODO : I think we can get xdis to support the dis api (python 3 version) by doing something like this there
from xdis import Bytecode, get_opcode
opc = get_opcode(PYTHON_VERSION_TRIPLE, IS_PYPY)
Bytecode = functools.partial(Bytecode, opc=opc)
import six
def _dis_to_text(co):
return Bytecode(co).dis()
def print_diff(original, uncompyled):
"""
Try and display a pretty html line difference between the original and
uncompyled code and bytecode if elinks and BeautifulSoup are installed
otherwise just show the diff.
:param original: Text describing the original code object.
:param uncompyled: Text describing the uncompyled code object.
"""
original_lines = original.split("\n")
uncompyled_lines = uncompyled.split("\n")
args = original_lines, uncompyled_lines, "original", "uncompyled"
try:
from bs4 import BeautifulSoup
diff = difflib.HtmlDiff().make_file(*args)
diff = BeautifulSoup(diff, "html.parser")
diff.select_one('table[summary="Legends"]').extract()
except ImportError:
print("\nTo display diff highlighting run:\n pip install BeautifulSoup4")
diff = difflib.HtmlDiff().make_table(*args)
with tempfile.NamedTemporaryFile(delete=False) as f:
f.write(str(diff).encode("utf-8"))
try:
print()
html = subprocess.check_output(
["elinks", "-dump", "-no-references", "-dump-color-mode", "1", f.name]
).decode("utf-8")
print(html)
except:
print("\nFor side by side diff install elinks")
diff = difflib.Differ().compare(original_lines, uncompyled_lines)
print("\n".join(diff))
finally:
os.unlink(f.name)
def are_instructions_equal(i1, i2):
"""
Determine if two instructions are approximately equal,
ignoring certain fields which we allow to differ, namely:
* code objects are ignore (should probably be checked) due to address
* line numbers
:param i1: left instruction to compare
:param i2: right instruction to compare
:return: True if the two instructions are approximately equal, otherwise False.
"""
result = (
1 == 1
and i1.opname == i2.opname
and i1.opcode == i2.opcode
and i1.arg == i2.arg
# ignore differences due to code objects
# TODO : Better way of ignoring address
and (i1.argval == i2.argval or "<code object" in str(i1.argval))
# TODO : Should probably recurse to check code objects
and (i1.argrepr == i2.argrepr or "<code object" in i1.argrepr)
and i1.offset == i2.offset
# ignore differences in line numbers
# and i1.starts_line
and i1.is_jump_target == i2.is_jump_target
)
return result
def are_code_objects_equal(co1, co2):
"""
Determine if two code objects are approximately equal,
see are_instructions_equal for more information.
:param i1: left code object to compare
:param i2: right code object to compare
:return: True if the two code objects are approximately equal, otherwise False.
"""
instructions1 = Bytecode(co1)
instructions2 = Bytecode(co2)
for opcode1, opcode2 in zip(instructions1, instructions2):
if not are_instructions_equal(opcode1, opcode2):
return False
return True
def validate_uncompyle(text, mode="exec"):
"""
Validate decompilation of the given source code.
:param text: Source to validate decompilation of.
"""
original_code = compile(text, "<string>", mode)
original_dis = _dis_to_text(original_code)
original_text = text
deparsed = code_deparse(
original_code, out=six.StringIO(), version=PYTHON_VERSION_TRIPLE, compile_mode=mode
)
uncompyled_text = deparsed.text
uncompyled_code = compile(uncompyled_text, "<string>", "exec")
if not are_code_objects_equal(uncompyled_code, original_code):
uncompyled_dis = _dis_to_text(uncompyled_text)
def output(text, dis):
width = 60
return "\n\n".join(
[
" SOURCE CODE ".center(width, "#"),
text.strip(),
" BYTECODE ".center(width, "#"),
dis,
]
)
original = output(original_text, original_dis)
uncompyled = output(uncompyled_text, uncompyled_dis)
print_diff(original, uncompyled)
assert "original" == "uncompyled"

View File

@@ -1,3 +1,3 @@
pytest
flake8
six
pytest==3.2.5 # for 2.7 < PYTHON_VERSION <= 3.2 use pytest 2.9.2; for 3.1 2.10
hypothesis

View File

@@ -1,9 +1,2 @@
# Pick up stuff from setup.py
hypothesis==2.0.0
pytest
-e .
Click~=7.0
xdis >= 6.1.0, < 6.2.0
configobj~=5.0.6
setuptools

View File

@@ -1,71 +0,0 @@
#!/usr/bin/env python
import sys
import setuptools
"""Setup script for the 'uncompyle6' distribution."""
SYS_VERSION = sys.version_info[0:2]
if SYS_VERSION < (3, 6):
mess = "Python Release 3.6 .. 3.12 are supported in this code branch."
if (2, 4) <= SYS_VERSION <= (2, 7):
mess += (
"\nFor your Python, version %s, use the python-2.4 code/branch."
% sys.version[0:3]
)
if SYS_VERSION >= (3, 6):
mess += (
"\nFor your Python, version %s, use the master code/branch."
% sys.version[0:3]
)
if (3, 0) >= SYS_VERSION < (3, 3):
mess += (
"\nFor your Python, version %s, use the python-3.0-to-3.2 code/branch."
% sys.version[0:3]
)
if (3, 3) >= SYS_VERSION < (3, 6):
mess += (
"\nFor your Python, version %s, use the python-3.3-to-3.5 code/branch."
% sys.version[0:3]
)
elif SYS_VERSION < (2, 4):
mess += (
"\nThis package is not supported for Python version %s." % sys.version[0:3]
)
print(mess)
raise Exception(mess)
from __pkginfo__ import (
__version__,
author,
author_email,
classifiers,
entry_points,
install_requires,
license,
long_description,
modname,
py_modules,
short_desc,
web,
zip_safe,
)
setuptools.setup(
author=author,
author_email=author_email,
classifiers=classifiers,
description=short_desc,
entry_points=entry_points,
install_requires=install_requires,
license=license,
long_description=long_description,
long_description_content_type="text/x-rst",
name=modname,
packages=setuptools.find_packages(),
py_modules=py_modules,
test_suite="nose.collector",
url=web,
version=__version__,
zip_safe=zip_safe,
)

View File

@@ -1,58 +1,11 @@
[bdist_rpm]
release = 0
packager = rocky <rb@dustyfeet.com
doc_files = README.rst
ChangeLog
COPYING
DECOMPYLE-2.4-CHANGELOG.txt
HISTORY.md
HOW-TO_REPORT-A-BUG.md
NEWS.md
release = 1
packager = Mysterie <kajusska@gmail.com>
doc_files = README
# CHANGES.txt
# USAGE.txt
# doc/
# examples/
[bdist_wheel]
universal = no
[metadata]
description_file = README.rst
licences_files = COPYING
[flake8]
# max-line-length setting: NO we do not want everyone writing 120-character lines!
# We are setting the maximum line length big here because there are longer
# lines allowed by black in some cases that are forbidden by flake8. Since
# black has the final say about code formatting issues, this setting is here to
# make sure that flake8 doesn't fail the build on longer lines allowed by
# black.
max-line-length = 120
max-complexity = 12
select = E,F,W,C,B,B9
ignore =
# E123 closing bracket does not match indentation of opening bracket's line
E123
# E203 whitespace before ':' (Not PEP8 compliant, Python Black)
E203
# E501 line too long (82 > 79 characters) (replaced by B950 from flake8-bugbear,
# https://github.com/PyCQA/flake8-bugbear)
E501
# W503 line break before binary operator (Not PEP8 compliant, Python Black)
W503
# W504 line break after binary operator (Not PEP8 compliant, Python Black)
W504
# C901 function too complex - since many of zz9 functions are too complex with a lot
# of if branching
C901
# module level import not at top of file. This is too restrictive. Can't even have a
# docstring higher.
E402
per-file-ignores =
# These are config files. The `c` variable them is injected not defined.
pow/ansible/roles/jupyterhub/templates/jupyterhub_config*.py:F821
# Ignore some errors in files that are stolen from other projects to avoid lots
# of merge problems later .
pow/ansible/roles/webtier/files/supervisor_httpgroupok.py:E126,E128,E222,E225,E226,E261,E301,E302,E305,F841,E201,E202
silhouette/src/silhouette/gprof2dot.py:E711,E713,E741,F401
# Ignore undefined name errors in "expectation" test Python code.
# These files get exec'd in an environment that defines the variables.
server/tests/files/expectations/*.py:F821
universal=1

View File

@@ -1,65 +1,29 @@
#!/usr/bin/env python
"""Setup script for the 'uncompyle6' distribution."""
import sys
import setuptools
from __pkginfo__ import \
author, author_email, install_requires, \
license, long_description, classifiers, \
entry_points, modname, py_modules, \
short_desc, VERSION, web, \
zip_safe
SYS_VERSION = sys.version_info[0:2]
if not ((3, 3) <= SYS_VERSION < (3, 6)):
mess = "Python Release 3.3 .. 3.5 are supported in this code branch."
if (2, 4) <= SYS_VERSION <= (2, 7):
mess += (
"\nFor your Python, version %s, use the python-2.4 code/branch."
% sys.version[0:3]
)
if SYS_VERSION >= (3, 6):
mess += (
"\nFor your Python, version %s, use the master code/branch."
% sys.version[0:3]
)
if (3, 0) >= SYS_VERSION < (3, 3):
mess += (
"\nFor your Python, version %s, use the python-3.0-to-3.2 code/branch."
% sys.version[0:3]
)
elif SYS_VERSION < (2, 4):
mess += (
"\nThis package is not supported for Python version %s." % sys.version[0:3]
)
print(mess)
raise Exception(mess)
from __pkginfo__ import (
__version__,
author,
author_email,
classifiers,
entry_points,
install_requires,
license,
long_description,
modname,
py_modules,
short_desc,
web,
zip_safe,
)
setuptools.setup(
author=author,
author_email=author_email,
classifiers=classifiers,
description=short_desc,
entry_points=entry_points,
install_requires=install_requires,
license=license,
long_description=long_description,
long_description_content_type="text/x-rst",
name=modname,
packages=setuptools.find_packages(),
py_modules=py_modules,
test_suite="nose.collector",
url=web,
version=__version__,
zip_safe=zip_safe,
)
from setuptools import setup, find_packages
setup(
author = author,
author_email = author_email,
classifiers = classifiers,
description = short_desc,
entry_points = entry_points,
install_requires = install_requires,
license = license,
long_description = long_description,
name = modname,
packages = find_packages(),
py_modules = py_modules,
test_suite = 'nose.collector',
url = web,
tests_require = ['nose>=1.0'],
version = VERSION,
zip_safe = zip_safe)

8
test/.gitignore vendored
View File

@@ -1,8 +0,0 @@
/.coverage
/.python-version
/nohup.out
/pycdc
/test_pycdc_tests.sh
/test_uncompyle2.py
/test_unpy33.py
/test_unpy37.py

View File

@@ -1,101 +1,62 @@
PHONY=check clean dist distclean test test-unit test-functional rmChangeLog clean_pyc nosetests \
check-bytecode-1.0 check-bytecode-1.1 check-bytecode-1.2 check-bytecode-1.3 \
check-bytecode-1 check-bytecode-1.4 check-bytecode-1.5 check-bytecode-1.6 \
check-bytecode-2 check-bytecode-3 check-bytecode-3-short \
check-bytecode-2.2 check-byteocde-2.3 check-bytecode-2.4 \
check-short check-2.6 check-2.7 check-3.0 check-3.1 check-3.2 check-3.3 \
check-3.4 check-3.5 check-3.6 check-3.7 check-5.6 5.6 5.8 \
grammar-coverage-2.5 grammar-coverage-2.6 grammar-coverage-2.7 \
grammar-coverage-3.1 grammar-coverage-3.2 grammar-coverage-3.3 \
grammar-coverage-3.4 grammar-coverage-3.5 grammar-coverage-3.6 \
grammar-coverage-3.7
PHONY=check clean dist distclean test test-unit test-functional rmChangeLog clean_pyc nosetests
GIT2CL ?= git2cl
PYTHON ?= python
PYTHON_VERSION = $(shell $(PYTHON) -V 2>&1 | cut -d ' ' -f 2 | cut -d'.' -f1,2 | head -1)
PYTHON_VERSION = $(shell $(PYTHON) -V | cut -d ' ' -f 2 | cut -d'.' -f1,2)
NATIVE_CHECK = check-$(PYTHON_VERSION)
# Set COMPILE='--compile' to force compilation before check
COMPILE ?=
COVER_DIR=../tmp/grammar-cover
# Run short tests
check-short:
@$(PYTHON) -V && PYTHON_VERSION=`$(PYTHON) -V 2>&1 | cut -d ' ' -f 2 | cut -d'.' -f1,2` | head -1; \
$(MAKE) check-bytecode-${PYTHON_VERSION}
@$(PYTHON) -V && PYTHON_VERSION=`$(PYTHON) -V 2>&1 | cut -d ' ' -f 2 | cut -d'.' -f1,2`; \
$(MAKE) check-bytecode
# Run all tests
check:
$(MAKE) check-$(PYTHON_VERSION)
@$(PYTHON) -V && PYTHON_VERSION=`$(PYTHON) -V 2>&1 | cut -d ' ' -f 2 | cut -d'.' -f1,2`; \
$(MAKE) check-$$PYTHON_VERSION
#: Run working tests from Python 2.6 or 2.7
check-2.6 check-2.7: check-bytecode-2 check-bytecode-3 check-bytecode-1 check-native-short
check-2.4 check-2.5 check-2.6 check-2.7: check-bytecode-2 check-bytecode-3 check-bytecode-1 check-2.7-ok
#: Run working tests from Python 3.0
#: Run working tests from Python 3.1
check-3.0: check-bytecode
$(PYTHON) test_pythonlib.py --bytecode-3.0-run --verify-run
$(PYTHON) test_pythonlib.py --bytecode-3.0 --syntax-verify $(COMPILE)
@echo Python 3.0 testing not done yet
#: Run working tests from Python 3.1
check-3.1: check-bytecode
$(PYTHON) test_pythonlib.py --bytecode-3.1-run --verify-run
$(PYTHON) test_pythonlib.py --bytecode-3.1 --syntax-verify $(COMPILE)
$(PYTHON) test_pythonlib.py --bytecode-3.1 --weak-verify $(COMPILE)
#: Run working tests from Python 3.2
check-3.2: check-bytecode
$(PYTHON) test_pythonlib.py --bytecode-3.2-run --verify-run
$(PYTHON) test_pythonlib.py --bytecode-3.2 --syntax-verify $(COMPILE)
$(PYTHON) test_pythonlib.py --bytecode-3.2 --weak-verify $(COMPILE)
#: Run working tests from Python 3.3
check-3.3: check-bytecode
$(PYTHON) test_pythonlib.py --bytecode-3.3-run --verify-run
$(PYTHON) test_pythonlib.py --bytecode-3.3 --syntax-verify $(COMPILE)
$(PYTHON) test_pythonlib.py --bytecode-3.3 --weak-verify $(COMPILE)
#: Run working tests from Python 3.4
check-3.4: check-bytecode check-3.4-ok check-2.7-ok
$(PYTHON) test_pythonlib.py --bytecode-3.4-run --verify-run
$(PYTHON) test_pythonlib.py --bytecode-3.4 --syntax-verify $(COMPILE)
$(PYTHON) test_pythonlib.py --bytecode-3.4 --weak-verify $(COMPILE)
#: Run working tests from Python 3.5
check-3.5: check-bytecode
$(PYTHON) test_pythonlib.py --bytecode-3.5-run --verify-run
$(PYTHON) test_pythonlib.py --bytecode-3.5 --syntax-verify $(COMPILE)
$(PYTHON) test_pythonlib.py --bytecode-3.5 --verify $(COMPILE)
#: Run working tests from Python 3.6
check-3.6: check-bytecode
$(PYTHON) test_pythonlib.py --bytecode-3.6-run --verify-run
$(PYTHON) test_pythonlib.py --bytecode-3.6 --syntax-verify $(COMPILE)
#: Run working tests from Python 3.7
check-3.7: check-bytecode
$(PYTHON) test_pythonlib.py --bytecode-3.7-run --verify-run
$(PYTHON) test_pythonlib.py --bytecode-3.7 --syntax-verify $(COMPILE)
check-pypy37: check-bytecode
$(PYTHON) test_pythonlib.py --bytecode-pypy37 --verify-run
#: Run working tests from Python 3.8
check-3.8: check-bytecode
$(PYTHON) test_pythonlib.py --bytecode-3.8-run --verify-run
$(PYTHON) test_pythonlib.py --bytecode-3.8 --syntax-verify $(COMPILE)
check-3.9: check-bytecode
@echo "Note that we do not support decompiling Python 3.9 bytecode - no 3.9 tests run"
check-3.10: check-bytecode
@echo "Note that we do not support decompiling Python 3.10 bytecode - no 3.10 tests run"
# FIXME
#: this is called when running under pypy3.5-5.8.0, pypy2-5.6.0, pypy3.6-7.3.0 or pypy3.8-7.3.7
5.8 5.6 7.3:
$(PYTHON) test_pythonlib.py --bytecode-3.6 --verify $(COMPILE)
#: Check deparsing only, but from a different Python version
check-disasm:
$(PYTHON) dis-compare.py
#: Check deparsing bytecode 1.x only
check-bytecode-1: check-bytecode-1.0 check-bytecode-1.1 check-bytecode-1.2 check-bytecode-1.3 check-bytecode-1.4 check-bytecode-1.5 check-bytecode-1.6
check-bytecode-1:
$(PYTHON) test_pythonlib.py --bytecode-1.5
#: Check deparsing bytecode 2.x only
check-bytecode-2:
@@ -104,70 +65,19 @@ check-bytecode-2:
--bytecode-2.5 --bytecode-2.6 --bytecode-2.7 --bytecode-pypy2.7
#: Check deparsing bytecode 3.x only
# check-bytecode-3:
# $(PYTHON) test_pythonlib.py \
# --bytecode-3.0 \
# --bytecode-3.1 --bytecode-3.2 --bytecode-3.3 \
# --bytecode-3.4 --bytecode-3.5 --bytecode-3.6 \
# --bytecode-3.7 \
# --bytecode-pypy3.2 --bytecode-pypy3.6 --bytecode-3.8
# FIXME: Until we shaked out problems with xdis...
check-bytecode-3:
$(PYTHON) test_pythonlib.py \
--bytecode-3.3 --bytecode-3.4 --bytecode-3.5 --bytecode-3.6 \
--bytecode-3.7 --bytecode-3.8
$(PYTHON) test_pythonlib.py --bytecode-3.0 \
--bytecode-3.1 --bytecode-3.2 --bytecode-3.3 \
--bytecode-3.4 --bytecode-3.5 --bytecode-pypy3.2
#: Check deparsing on selected bytecode 3.x
check-bytecode-3-short:
$(PYTHON) test_pythonlib.py \
--bytecode-3.4 --bytecode-3.5 --bytecode-3.6
#: Check deparsing bytecode on all Python 2 and Python 3 versions
#: Check deparsing bytecode that works running Python 2 and Python 3
check-bytecode: check-bytecode-3
$(PYTHON) test_pythonlib.py \
--bytecode-1.0 --bytecode-1.1 --bytecode-1.2 \
--bytecode-1.3 --bytecode-1.4 --bytecode-1.5 \
--bytecode-2.2 --bytecode-2.3 --bytecode-2.4 \
--bytecode-2.1 --bytecode-2.2 --bytecode-2.3 --bytecode-2.4 \
--bytecode-2.5 --bytecode-2.6 --bytecode-2.7 \
--bytecode-pypy2.7
--bytecode-pypy2.7 --bytecode-1
#: Check deparsing bytecode on selected Python 2 and Python 3 versions
check-bytecode-short: check-bytecode-3-short
$(PYTHON) test_pythonlib.py \
--bytecode-2.6 --bytecode-2.7 --bytecode-pypy2.7
#: Check deparsing bytecode 1.0 only
check-bytecode-1.0:
$(PYTHON) test_pythonlib.py --bytecode-1.0
#: Check deparsing bytecode 1.1 only
check-bytecode-1.1:
$(PYTHON) test_pythonlib.py --bytecode-1.1
#: Check deparsing bytecode 1.2 only
check-bytecode-1.2:
$(PYTHON) test_pythonlib.py --bytecode-1.2
#: Check deparsing bytecode 1.3 only
check-bytecode-1.3:
$(PYTHON) test_pythonlib.py --bytecode-1.3
#: Check deparsing bytecode 1.4 only
check-bytecode-1.4:
$(PYTHON) test_pythonlib.py --bytecode-1.4
#: Check deparsing bytecode 1.5 only
check-bytecode-1.5:
$(PYTHON) test_pythonlib.py --bytecode-1.5
#: Check deparsing bytecode 1.6 only
check-bytecode-1.6:
$(PYTHON) test_pythonlib.py --bytecode-1.6
#: Check deparsing Python 2.1
check-bytecode-2.1:
$(PYTHON) test_pythonlib.py --bytecode-2.1
@@ -182,193 +92,80 @@ check-bytecode-2.3:
#: Check deparsing Python 2.4
check-bytecode-2.4:
$(PYTHON) test_pythonlib.py --bytecode-2.4-run --verify-run
$(PYTHON) test_pythonlib.py --bytecode-2.4
#: Check deparsing Python 2.5
check-bytecode-2.5:
$(PYTHON) test_pythonlib.py --bytecode-2.5-run --verify-run
$(PYTHON) test_pythonlib.py --bytecode-2.5
#: Check deparsing Python 2.6
check-bytecode-2.6:
$(PYTHON) test_pythonlib.py --bytecode-2.6-run --verify-run
$(PYTHON) test_pythonlib.py --bytecode-2.6 --syntax-verify
$(PYTHON) test_pythonlib.py --bytecode-2.6
#: Check deparsing Python 2.7
check-bytecode-2.7:
$(PYTHON) test_pythonlib.py --bytecode-2.7-run --verify-run
$(PYTHON) test_pythonlib.py --bytecode-2.7 --syntax-verify
#: Get grammar coverage for Python 2.4
grammar-coverage-2.4:
-rm $(COVER_DIR)/spark-grammar-24.cover
SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-2.4.cover $(PYTHON) test_pythonlib.py --bytecode-2.4
SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-2.4.cover $(PYTHON) test_pyenvlib.py --2.4.6 --max= 800
#: Get grammar coverage for Python 2.5
grammar-coverage-2.5:
-rm $(COVER_DIR)/spark-grammar-2.5.cover || true
SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-2.5.cover $(PYTHON) test_pythonlib.py --bytecode-2.5
SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-2.5.cover $(PYTHON) test_pyenvlib.py --2.5.6 --max=800
#: Get grammar coverage for Python 2.6
grammar-coverage-2.6:
-rm $(COVER_DIR)/spark-grammar-2.6.cover || true
SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-2.6.cover $(PYTHON) test_pythonlib.py --bytecode-2.6
SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-2.6.cover $(PYTHON) test_pyenvlib.py --2.6.9 --max=800
#: Get grammar coverage for Python 2.7
grammar-coverage-2.7:
-rm $(COVER_DIR)/spark-grammar-2.7.cover || true
SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-2.7.cover $(PYTHON) test_pythonlib.py --bytecode-2.7
SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-2.7.cover $(PYTHON) test_pyenvlib.py --2.7.16 --max=600
#: Get grammar coverage for Python 3.0
grammar-coverage-3.0:
-rm $(COVER_DIR)/spark-grammar-30.cover
SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-30.cover $(PYTHON) test_pythonlib.py --bytecode-3.1
SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-30.cover $(PYTHON) test_pyenvlib.py --3.0.1
#: Get grammar coverage for Python 3.1
grammar-coverage-3.1:
-rm $(COVER_DIR)/spark-grammar-3.1.cover
SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.1.cover $(PYTHON) test_pythonlib.py --bytecode-3.1
SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.1.cover $(PYTHON) test_pyenvlib.py --3.1.5
#: Get grammar coverage for Python 3.2
grammar-coverage-3.2:
-rm $(COVER_DIR)/spark-grammar-3.2.cover || true
SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.2.cover $(PYTHON) test_pythonlib.py --bytecode-3.2
SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.2.cover $(PYTHON) test_pyenvlib.py --3.2.6
#: Get grammar coverage for Python 3.3
grammar-coverage-3.3:
-rm $(COVER_DIR)/spark-grammar-3.3.cover || true
SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.3.cover $(PYTHON) test_pythonlib.py --bytecode-3.3
SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.3.cover $(PYTHON) test_pyenvlib.py --3.3.7 --max=800
#: Get grammar coverage for Python 3.4
grammar-coverage-3.4:
-rm $(COVER_DIR)/spark-grammar-3.4.cover || true
SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.4.cover $(PYTHON) test_pythonlib.py --bytecode-3.4
SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.4.cover $(PYTHON) test_pyenvlib.py --3.4.8 --max=800
#: Get grammar coverage for Python 3.5
grammar-coverage-3.5:
rm $(COVER_DIR)/spark-grammar-3.5.cover || /bin/true
SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.5.cover $(PYTHON) test_pythonlib.py --bytecode-3.5
SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.5.cover $(PYTHON) test_pyenvlib.py --3.5.5 --max=450
#: Get grammar coverage for Python 3.6
grammar-coverage-3.6:
rm $(COVER_DIR)/spark-grammar-3.6.cover || /bin/true
SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.6.cover $(PYTHON) test_pythonlib.py --bytecode-3.6
SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.6.cover $(PYTHON) test_pyenvlib.py --3.6.8 --max=280
#: Get grammar coverage for Python 3.7
grammar-coverage-3.7:
rm $(COVER_DIR)/spark-grammar-3.7.cover || /bin/true
SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.7.cover $(PYTHON) test_pyenvlib.py --3.7.3 --max=500
$(PYTHON) test_pythonlib.py --bytecode-2.7 --verify
#: Check deparsing Python 3.0
check-bytecode-3.0:
$(PYTHON) test_pythonlib.py --bytecode-3.0-run --verify-run
$(PYTHON) test_pythonlib.py --bytecode-3.0 --syntax-verify
$(PYTHON) test_pythonlib.py --bytecode-3.0
#: Check deparsing Python 3.1
check-bytecode-3.1:
$(PYTHON) test_pythonlib.py --bytecode-3.1-run --verify-run
$(PYTHON) test_pythonlib.py --bytecode-3.1 --syntax-verify
$(PYTHON) test_pythonlib.py --bytecode-3.1
#: Check deparsing Python 3.2
check-bytecode-3.2:
$(PYTHON) test_pythonlib.py --bytecode-3.2-run --verify-run
$(PYTHON) test_pythonlib.py --bytecode-3.2 --syntax-verify
$(PYTHON) test_pythonlib.py --bytecode-3.2
#: Check deparsing Python 3.3
check-bytecode-3.3:
$(PYTHON) test_pythonlib.py --bytecode-3.3-run --verify-run
$(PYTHON) test_pythonlib.py --bytecode-3.3 --syntax-verify
$(PYTHON) test_pythonlib.py --bytecode-3.3
#: Check deparsing Python 3.4
check-bytecode-3.4:
$(PYTHON) test_pythonlib.py --bytecode-3.4-run --verify-run
$(PYTHON) test_pythonlib.py --bytecode-3.4 --syntax-verify
$(PYTHON) test_pythonlib.py --bytecode-3.4
#: Check deparsing Python 3.5
check-bytecode-3.5:
$(PYTHON) test_pythonlib.py --bytecode-3.5-run --verify-run
$(PYTHON) test_pythonlib.py --bytecode-3.5 --syntax-verify
$(PYTHON) test_pythonlib.py --bytecode-3.5
#: Check deparsing Python 3.6
check-bytecode-3.6:
$(PYTHON) test_pythonlib.py --bytecode-3.6-run --verify-run
$(PYTHON) test_pythonlib.py --bytecode-3.6 --syntax-verify
#: Check deparsing Python 3.7
check-bytecode-3.7:
$(PYTHON) test_pythonlib.py --bytecode-3.7-run --verify-run
$(PYTHON) test_pythonlib.py --bytecode-3.7 --syntax-verify
#: Check deparsing Python 3.8
check-bytecode-3.8:
$(PYTHON) test_pythonlib.py --bytecode-3.8-run --verify-run
$(PYTHON) test_pythonlib.py --bytecode-3.8 --syntax-verify
$(PYTHON) test_pythonlib.py --bytecode-3.6
#: short tests for bytecodes only for this version of Python
check-native-short:
$(PYTHON) test_pythonlib.py --bytecode-$(PYTHON_VERSION) --syntax-verify $(COMPILE)
$(PYTHON) test_pythonlib.py --bytecode-$(PYTHON_VERSION)-run --verify-run $(COMPILE)
$(PYTHON) test_pythonlib.py --bytecode-$(PYTHON_VERSION) --verify $(COMPILE)
#: Run longer Python 2.6's lib files known to be okay
check-2.6-ok:
$(PYTHON) test_pythonlib.py --ok-2.6 --syntax-verify $(COMPILE)
$(PYTHON) test_pythonlib.py --ok-2.6 --verify $(COMPILE)
#: Run longer Python 2.7's lib files known to be okay
check-2.7-ok:
$(PYTHON) test_pythonlib.py --ok-2.7 --syntax-verify $(COMPILE)
$(PYTHON) test_pythonlib.py --ok-2.7 --verify $(COMPILE)
#: Run longer Python 3.2's lib files known to be okay
check-3.2-ok:
$(PYTHON) test_pythonlib.py --ok-3.2 --syntax-verify $(COMPILE)
$(PYTHON) test_pythonlib.py --ok-3.2 --verify $(COMPILE)
#: Run longer Python 3.4's lib files known to be okay
check-3.4-ok:
$(PYTHON) test_pythonlib.py --ok-3.4 --syntax-verify $(COMPILE)
$(PYTHON) test_pythonlib.py --ok-3.4 --verify $(COMPILE)
#: PyPy of some sort. E.g. [PyPy 5.0.1 with GCC 4.8.4]
# Skip for now
2.6:
#: PyPy 5.0.x with Python 2.7 ...
pypy-2.7 5.0 5.3 6.0:
pypy-2.7 5.0 5.3:
$(PYTHON) test_pythonlib.py --bytecode-pypy2.7 --verify
#: PyPy 2.4.x with Python 3.2 ...
pypy-3.2 2.4:
$(PYTHON) test_pythonlib.py --bytecode-pypy3.2 --verify
#: PyPy 5.0.x with Python 3.6.1 ...
check-bytecode-pypy3.6: 7.1
7.1:
$(PYTHON) test_pythonlib.py --bytecode-pypy3.6-run --verify-run
$(PYTHON) test_pythonlib.py --bytecode-pypy3.6 --verify
#: PyPy 5.0.x with Python 3.6.9
check-bytecode-pypy3.6: 7.2 7.3
7.2:
$(PYTHON) test_pythonlib.py --bytecode-pypy3.6-run --verify-run
$(PYTHON) test_pythonlib.py --bytecode-pypy3.6 --verify
7.3:
$(PYTHON) test_pythonlib.py --bytecode-pypy3.7 --verify-run
# Pyston 2.3
2.3: check-3.8
clean: clean-py-dis clean-dis clean-unverified
clean-dis:

View File

@@ -1,49 +1,16 @@
#!/usr/bin/env python
""" Trivial helper program to byte compile and uncompile the bytecode file.
""" Trivial helper program to bytecompile and run an uncompile
"""
import os, sys, py_compile
from xdis.version_info import version_tuple_to_str, PYTHON_VERSION_TRIPLE
if len(sys.argv) < 2:
print("Usage: add-test.py [--run] *python-source*... [optimize-level]")
sys.exit(1)
assert 2 <= len(sys.argv) <= 4
assert len(sys.argv) >= 2
version = sys.version[0:3]
vers = sys.version_info[:2]
if sys.argv[1] in ("--run", "-r"):
suffix = "_run"
assert len(sys.argv) >= 3
py_source = sys.argv[2:]
i = 2
else:
suffix = ""
py_source = sys.argv[1:]
i = 1
try:
optimize = int(sys.argv[-1])
assert sys.argv >= i + 2
py_source = sys.argv[i:-1]
i = 2
except:
optimize = 2
for path in py_source:
for path in sys.argv[1:]:
short = os.path.basename(path)
if short.endswith(".py"):
short = short[: -len(".py")]
if hasattr(sys, "pypy_version_info"):
version = version_tuple_to_str(end=2, delimiter="")
bytecode = "bytecode_pypy%s%s/%spy%s.pyc" % (version, suffix, short, version)
if hasattr(sys, 'pypy_version_info'):
cfile = "bytecode_pypy%s/%s" % (version, short) + 'c'
else:
version = version_tuple_to_str(end=2)
bytecode = "bytecode_%s%s/%s.pyc" % (version, suffix, short)
print("byte-compiling %s to %s" % (path, bytecode))
if PYTHON_VERSION_TRIPLE >= (3, 2):
py_compile.compile(path, bytecode, optimize=optimize)
else:
py_compile.compile(path, bytecode)
if PYTHON_VERSION_TRIPLE >= (2, 6):
os.system("../bin/uncompyle6 -a -t %s" % bytecode)
cfile = "bytecode_%s/%s" % (version, short) + 'c'
print("byte-compiling %s to %s" % (path, cfile))
py_compile.compile(path, cfile)
if isinstance(version, str) or version >= (2, 6, 0):
os.system("../bin/uncompyle6 -a -t %s" % cfile)

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Some files were not shown because too many files have changed in this diff Show More