Justifying the conversion of a legacy project, that's still being maintained, is very difficult since the conversion doesn't necessarily bring anything tangible, without refactoring, especially if you have fixed dependencies. And, until python 3.7, the conversion would just make everything slower. That's something you never want on your justification list.
There are many apps written in the old Python nobody is going to rewrite in Python 3 despite many people may need them occasionally. In many cases they can be used for decades as long as there is a compatible interpreter available - there are many contexts where security fixes are not really necessary.
There also are platforms where new Python is not and not going to be available, e.g. DOS.
In case you look back at this comment, the author has posted in the root of the thread. BLUF: A proof of concept that python3 minus unicode strings could have been done in python2 with relative ease.
Many people dislike Python 3 syntax. I think this is a great project, and given the mishandling of Python2/3, I wish we had a more systematic way to introduce new features. I hope this succeeds.
The aspect that breaks existing code. They could very well, for example, create a print function that didn't break existing print statement. I think it was a childish move.
I mean, it was a pretty massive upgrade, and 2to3 exists for those who can't appreciate the value of a a function actually being composable instead of a wrongly-handled statement in the first place.
I've worked in and around Python for more than a decade, and I've yet to meet a single developer who didn't initially dig their heels in over `print` being changed, and then later realize that they were flat wrong to have dug their heels in because the new way is measurably better.
I think it was the correct move IMO. I find `print x` more confusing than `print(x)`. Same with the typeof statement in js. If something has an input, it should express that in some consistent way.
typeof in JavaScript is a unary operator, not a statement or a function. Which makes sense and squares with instanceof.
It would be bonkers for print to be a unary operator because it's just a side effect, and so I feel that your point in this situation is correct where it would not be in your other examples.
This isn't ideal as a version upgrade may also change the parameter type from a third party library, and these tools cannot detect it:
old.py:
bs = raw_input()
# Call a 3ps which originally accepts a bytestream, but now accepts a string
some_3rd_party_function(bs)
After python-modernize:
from six.moves import input
bs = input()
# Your code using bs as a bytestring still works
# Runtime Error:
# This now accepts a string, and you need to modify your code to deal with random TypeError popping from everywhere
some_3rd_party_function(bs)
...wrong, and probably also mythical. There's no excuse for having it as a language directive rather than a function call. I don't believe that any human who's spent even a few seconds thinking about it actually prefers the former over the latter.
Python 2 was my first language and I've spent years stewing over this horrible change. I just don't bother much with python now.
It's supposed to be a convenient, quick language for scripting. There was no good reason to force some academic consistency on the print call. It's extra keystrokes for no reason. It makes me sick every time I have to use the parentheses.
I do. There's a huge advantage that print as a construct has: print >>fp, ...
Yes, you can do the same thing with a kwarg on the print function, but it goes at the end of the line and ends up substantially messier in practice. This is the #1 reason that I still write Python 2 every single day; it makes a huge difference for code generation tasks.
Unfortunately, that only solves the case of printing strings with newlines, not the case where I might want to continue the line with a space instead, where I'd normally just add "," at the end and call it good. There are absolutely ways around their poor decision, but at the end of the day, the print function instead of statement is a bad fit for some use cases.
While it's nice to get rid of the weird special-case syntax and to be able to call print in expressions, I do prefer the old print statement syntactically (even after several seconds of thought) because it's easier to type.
Maybe in Python 4 we can have Nim's approach[1] (allowing both) and another decade of arguing about it.
I believe that was a joke. I mean he can actually dislike the operator but I'm sure he knows he doesn't have to use it and it also isn't what makes Python 3 different from Python 2 - it just has been added (in 12 years since Python 3.0 release) and nobody uses it in production at the moment anyway.
This is a bit unfair. If your language adopts a new feature, others on your team are likely to start using it even if you don't. You're allowed to feel a way about it.
But this is literally python 3 syntax. I'm absolutely not following. They adopted python 2 codebase and implemented `await`, `async` etc syntax in python 3. How is this any better than python 3 syntax-wise?
It would be much more helpful to make it possible to use Python 2 syntax with a Python 3 interpreter. Ideally as an extension for Python 3. That would allow people to incrementally port.
Python 3 syntax is mostly a superset of the 2 language, except for losing "print." The portability issues are around changed semantics of the main string type, changes to the C module API, and renaming of standard library components — rather than syntax. That's been my experience, anyway.
If you were starting a project now, you wouldn't choose this. It was done to prove a point that Python 2 could have gotten (nearly) all the Python 3 features without a hard/breaking change.
Their point has been made, but it's not like the Python community is going to unmake those changes. If you're starting a project, you should definitely use the current version of a language that is supported by a zillion developers, rather than picking a (practically experimental) fork of the old version.
Shrug. The last commit on the Python.org 2.7 branch was 3 months ago, per the release notes from the other thread today. It wasn't especially active before that. By their own plan, Python.org 2.7's "last commit age" will continue increasing forever.
This isn't a great metric for evaluating Python 2 forks, that's all.
The Python.org 2.7 branch is unmaintained so that’s hardly a strong point. That last commit, barring some exceptional circumstance, will be the last ever received by that branch.
According to the open issues, it doesn't currently support `async for`, nor is anything from 3.7 or 3.8 included (dataclasses, f-strings, breakpoint(), a number of performance improvements).
There hasn't been a main branch commit since python 2.7.17 was merged in ~6 months ago.
I guess it seems hard, but if you understand the basics it's just a matter of finding IO and encoding/decoding things appropriately. Python 2 str, unicode and Python 3 str are almost exactly the same as they ever were, and if you are doing binary IO bytes should work as expected.
It's not that big of a project, even for a moderately large codebase. If you think you can't get it done in a reasonable period of time feel free to hire me as a contractor and I'll knock it out.
The real problem in practice is all the places that relied on implicit bytes <-> unicode conversions. These are also likely to be broken on unusual inputs due to ASCII being the default encoding, rather than user locale... but that kind of thing is depressingly common even so.
there isn't an implicit bytes/unicode conversion in python2.
Do you mean calling unicode('some non unicode string')? That just uses system default encoding. ( sys.setdefaultencoding() ). Just find and replace them and slap in a .decode('UTF-8') or whatever your default encoding was in python2.
Grep for the strings encode, decode, unicode and just mechanically fix them one at a time by making the old implicit behavior explicit. How many times could you be doing that anyway? A few hundred? A thousand? You could even script this pretty reliably and just page through the diff you end up with to eyeball them one at a time.
I guess you might mean 'str' + u'unicodestr' or something, but again you can find these pretty easily by rooting out where the non-unicode strings are being produced and fixing the problem there. They are either literals or they are coming from IO or calls to str, right? Anyway, I've done this quite a few times and the main concern I've always had was trying to get the patch in place before people commit too much stuff for me to be able to merge the fixed up branch.
Of course, you could just do it little by little by taking out places that you are relying on systemdefaultencoding by monkeypatching the default decoding function in sys to log tracebacks whenever it is used, and then whacking them as they come up so you end up with properly handled and explicit unicode decoding before you move away from python2. I bet you could find and fix 95% of the cases in a day of effort.
('str' + u'unicodestr') involves an implicit conversion of the bytes operand on the left side to unicode, yes, so clearly Python 2 has it. And it's not the only such case- the fundamental problem is that the Python/C API for Python 2 implements this for the standard argument parsing functions. So basically any function implemented in native code that invokes PyArg_ParseTuple("s"), will also accept Unicode objects, and will implicitly encode them with ASCII. IIRC the reverse is true for "u". So those conversions happen every time you cross from Python into native - and all built-in data types, operators, and functions are native, as are large swaths of the standard library.
And yeah, u"" literals aren't hard to find, but the problem is when you get data flowing from different sources, so both sides are variables. For example, one is read from a text file, and another one comes from parsed JSON - so the former is raw bytes, and the latter is Unicode - and you need to combine them together. Like you said, the proper way to do this is to ensure that as soon as data crosses the I/O boundary, it should be of the correct type (i.e. unicode rather than bytes) - which, ironically, is exactly what Python 3 encourages with its changes. But it can be hard to find all such places - you have to actually audit every use of I/O one by one, because in Python 2, the code by itself doesn't always reflect whether it's supposed to be dealing with text or binary data.
So I'm back to my idea of monkeypatching the standard libraries to add logging to a file wherever it resorts to the default encoding or an implicit decode, and one by one fix them until the log file stops having stuff in it. After a couple of months of not seeing any you just assume you found them all, and you can safely bet that you'll find the last couple in a couple more months, but by then you have managed to move to Python3 a few months ago.
Print is not the only difference between python 2 and 3.
Strings, most things are generators now, exceptions are different, relative imports changed, division, etc. On top of that, moving to python 3 requires updating your dependencies.
If you're looking to continue running old codebases without testing and patching this isn't it. There are various incompatibilities with CPython 2.7, some noted in the readme, some noted in issues e.g. [0], and no doubt others unknown. It also doesn't appear to be actively maintained..
I think we need to recognize that most people/companies who stick to python 2 doesn't do it because they are being stubborn. There is a real judgment call involved. It might look strange from the outside, but maybe it really is daunting to migrate some code-bases to python 3.
In my experience, it's been a number of years since the judgement call was something other than “can we avoid paying for maintenance?”. Active projects had reasons 5+ years ago but by now it's really just a question of whether your team has a handle on technical debt management or is on an unsustainable feature treadmill.
Daunting? Sure, but it's also daunting jumping from branch to branch trying to find somebody to support your rusty old Python 2 installations. Or deciding to actively run on unsupported Python.
Modern software isn't fire and forget. Developers who have grown up around security exploits understand that. Higher management and FORTRAN77 programmers don't. To them it's developer busywork.
The real judgement being made is money and resources.
Leaving aside the bugfixes they've kept publishing for Python 2.7 (and it's rather massive standard library), how is your stack of ~100 external dependencies holding up? In my experience, most libraries are dropping Python 2 support too. Many dropped it a while ago.
Your locally running, toy projects will continue to run under Python 2 indefinitely. Large [esp network-facing] projects will start to fall apart.
Many large Python 2 projects come from age when it was not exactly practical to have ~100 external dependencies and thus do not have that. It was the time when one really though about whether having external dependency for this particular shiny thing is worth the deployment hassle involved.
Indeed. The main project I work on at the day gig is old enough that it's build on an in-house framework that predates the first public release of Django.
Python 2.7 is not dead because the PSF stopped supporting it. Thinking it is would be equivalent to thinking that C is dead because Bell Labs no longer supports it.
I predict that Python 2.7 will continue being supported by some capable organization for at least the next 10 years. As a lower bound, Red Hat has promised to continue supporting it for RHEL 8 customers until at least 2024.
This is a Tinkerbell argument, but it's accurate enough. Python 2 will exist in some form, while people believe in it enough. But as you say, that support is likely to be paid.
Many deployments rely on more than a supported cpython binary. External packages and the rest of the server stack. The Python community has largely dropped Python 2 support. If you don't need that, great. If you do... Then what?
You could pay people to support your entire stack. You could upgrade. It's that choice that makes Python 2 dead.
You can still run Windows 2000. Doesn't mean it's not dead.
> The real judgement being made is money and resources.
Unnecessary changes and the overall fragility of "modern" software takes money and other resources away from dealing with actual security issues.
There's no reason why things like a library of geospatial calculations (distance between two lat/lon pairs, etc) should need constant maintenance. They could be written once and then used for decades.
That's the biggest problem with the "old" ways of doing things, you simply don't know where you're vulnerable. Refusing to allocate resources to revisit old code, only to add new features is a recipe for disaster.
Upgrading to Python 3 does not fix this alone, but ignoring it, and the other dependencies that you might have, and that old Redhat 7.1 box running Linux 2.4 that's all probably fine, it's just card processing, right? Nothing else has changed.
The "modern" ethos is testing everything all the time. Yeah, it's a shed-load of extra stuff if it's new to your project, but it's only codifying things that you should be doing anyway, even if only seasonally.
Yes, there is occasional fragility, but that's more than offset —at least in my network-facing world— by the constant stream of security fixes across my stack.
Here’s my real world example of some Python 2 code which will never get ported to Python 3 (and the resulting Ycombinator heated discussion): https://news.ycombinator.com/item?id=21258527
Firms should only embark on NPV positive projects. It’s possible that the efficient gains from the new python features outweigh the cost, but tough to prove.
No it wasn't; it was the better part of a decade. Look, we all know Python 3 sucks (even if some poeple, unfortunately including the former Python maintainers, refuse to admit it), but this sort of blatant falsehood helps noone.
Good thing python didn't have a package management system where it was easy to take an external dependency on something that would then later be a monumental task to update. Or, even if you did do that, at least the automatic 2to3 upgrade tool was so well polished you wouldn't need to spend much time figuring out why it wasn't working.
Original author here. I built this a few years ago, and the main motivation at the time was that I'd heard people say that adding the new features in Python3 required breaking backwards compatibility, which I didn't believe. IMO the only feature that really required breaking backwards compatibility was the str/unicode consolidation and refactoring. This project was my way of proving that we could have gotten the other features that people tend to be most excited about (async/await, function annotations, new super(), etc) without breaking existing Python2 code. I think it was successful at that as a proof of concept.
It was a fun project; I learned a lot about how the CPython implementation works and have a lot of respect for the people that built it. It was surprisingly easy to implement Tauthon based off the work the core dev team did on Python3: https://www.naftaliharris.com/blog/nonlocal/
For what it's worth, I do believe that Python3 is a better language than Python2. We use Python 3.7 at my work (SentiLink) and we've had a good experience with it. (If you're starting a new project or can migrate, I'd recommend it). But I do think that the ~10 year saga of upgrading to Python3 from Python2 wasn't necessary when the main benefit was really the unicode refactoring.
I no longer maintain Tauthon personally but there are others who are excited about the project who occasionally add new features or bugfixes.
meanwhile exceptions just sob in a corner, forgotten.
the main benefit is actually sane exceptions. unicode is nice and all (i'm a native speaker of a non-english language) but it's a storm in a teacup IME.
I feel a bit like you’re trying to rewrite history here.
When you launched it, you called it “Python 2.8”. You posted it everywhere to gain traction, and didn’t rename it until the PSF and Guido got you by the ear, so to speak. There was no mention of “everything but str” or whatever, as far as I recall.
It was an outright (and hostile) attempt to fork - something that, going by your words here, I guess you now recognise as a mistake. I guess saying “I screwed up” is hard.
You're coloring the history with a lot more hostility than the reality. And the condescension is really unnecessary.
This matches the pattern of Python.org developers and python 3 aficionados being unnecessarily hostile and condescending to the concerns of Python 2 language users. You saw that in 2010; you saw it again in 2015; and you can see it in these threads today.
Can you explain the specific actions that were taken that didn't cater to, as you say, "python2 language users"? A 12 year migration timeline seems, to me, to be fairly lax.
I am definitely guilty of being "hostile and condescending" to people asking the Python.org developers to waste time supporting Python 2. Please just stop. This time would be better spent working on the packaging situation rather than relitigating the last 12 years.
If people still want to work on Python 2 projects that's fine and it's their choice, but it's time to just let the rest of us go our own way.
None of the things you mentioned are particularly difficult though. In fact you're still in the realm of changes that can be trivially automated with https://python-modernize.readthedocs.io/en/latest/fixers.htm..., the three issues you describe are the print, xrange_six, and classic_division fixers.
It's certainly possible that there are parts of the migration that would be tricky, a quick skim of the file didn't give me any obvious ones, but it's also huge and hard to read, so I very well could have missed something.
Most of the truly challenging things to migrate involved some combination of extension modules, heavy metaprogramming (eval/exec), and apis which change significantly between 2 and 3 (most of which are string related, but some libraries also decided to do backwards incompatible things)
Can you provide citations for your claims? The commentary in the github renaming issue from 2016(0) seems pretty jovial and has the tone of "oh, actually we should change that, oops" rather than "we WILL replace python!". The comments by guido and harris reinforce this perspective:
> gvanrossum commented on Dec 10, 2016
> Since I was asked: The project's name (and its binary name) need to
> change. They are misleading. The rest looks acceptable according to
> Python's license. This is not an endorsement (far from it).
> naftaliharris commented on Dec 10, 2016
> I don't mind renaming this project. Any other suggestions for good
> names? I personally like "Pythonesque (/usr/bin/pesque)" the best so
> far, thanks @dbohdan! :-)
>
> @VanL, not that I'm necessarily picking that, but would a name like
> that be acceptable?
When you're in a community or a space, actions that are unintentionally hostile towards that community or space, can be seen as intentionally hostile by the members, and if you only hear about it second-hand, or you spend a lot of time around the group, that belief can be reinforced through the discussions and gripes the group has about it(1).
(0): https://github.com/naftaliharris/tauthon/issues/47
(1): That's not to say there aren't bona fide intentionally hostile actions that happen, but rather that's how actions that aren't intended to be hostile can be remembered and percieved as such.
It was all very disingenuous. It was obvious to everyone, after 8 years of flames, that such a move would have been incendiary. Naftali played dumb only after he was called out on it.
I'm not saying it was entirely his fault - certain widely-heard voices in the community had been advocating for this to happen, in practice, for several months; he saw an opportunity and went for it. I just object to the rewrite of history to justify the mistakes of the past.
> It was all very disingenuous. It was obvious to everyone, after 8 years of flames, that such a move would have been incendiary. Naftali played dumb only after he was called out on it.
Have you considered that not everyone who forks something participates in the original community?
> I just object to the rewrite of history to justify the mistakes of the past.
So far there has been no evidence for the stated claim, just supposition and rumour. So as-is there's no reason for anyone here to believe that "history is being rewritten" aside from easily-mistaken word of mouth.
These days most discussions over the internet happen via the written word, so it's difficult to believe that you can't find records from IRC, Github, or Email to support your contention that it was hostile.
Do you have any references that support the level of invective you are displaying? I feel you are projecting a great deal of ill will on the original author that I, personally, didn't feel or see.
I wouldn't call it a "hostile" fork. Imagine what motivated him to fork in the first place. He and many others were between a rock and a hard place, and a fork was a possible way out.
At the time, the author stated:
> I picked [the name "Python 2.8"] initially since when talking with friends about this project it conveyed pretty darn immediately what the project is and does. I'd be very keen to hear people's suggestions for alternate names!
This was no mistake or screw up at all. In fact, the project served his purpose for the time, and even though he's moved on, there are others who like it enough to maintain or improve it.
A fork is (or can be) a healthy, natural thing that happens.
As a general question, why is forking a project perceived as hostile? I always thought that forking an open source project was something that was encouraged if one's proposed contributions were not incorporated into the original project.
First off, cool project Naftali! One of the issues that ultimately became more common as Python2 became EOL is that a lot of libraries began to release only wheels for Python 3.x variants. When there's C/C++ extensions this means that even running a compatible fork wouldn't suffice.
Do you think it would have been feasible to make wheels compiled against the python 3.x C APIs also compatible with Tauthon?
I think there is some demand for legacy Python 2 to be maintained, even if the maintenance is limited only to security fixes, and I think it would be good if some company paid you or someone else a full time salary to continue maintaining this Python 2 fork.
It’s a tragedy of the commons: A lot of companies making good money are still using Python 2, but none of them are willing to pay programmers to maintain an open source currently maintained Python 2 fork.
ActiveState is offering commercial support for Python 2, for those who need it. But places that are so conservative as to need it, are also precisely the ones that will want vanilla Python rather than a fork.
I don't know what it supposed to prove. It was no secret that Unicode was what broke compatibility, the other changes were there to use that as an opportunity to fix warts that accumulated over years (like organization of system packages, print statement, division, classes etc. I don't believe Thauton fixed these things though)
It was no secret that everything else was portable, that's essentially all what python 2.7 was - backporting 3.x features to Python 2. It was a waste of resources for developers to maintain 2 forks of Python so that effort was stopped in 2015. There was 5 extra years to move application to Python 3.
5 years is frigging long time in computer terms, and your project felt still like giving f-you to the core developers for trying to improve the language. Trying to call the project Python 2.8 was very aggressive and would create a lot of confusion if Guido would accept that.
I'm glad you gave python 3 a try, I feel like the people that were against it didn't write any new application in it, and their experience was porting Python 2 code to 3. It can be very frustrating when Python 3 complains about bugs that Python 2 just ignored, but if you write a python 3 application from scratch you don't even notice the Unicode, and that was the goal.
Some of the changes I feel were a step too far though. Completely removing iter{items,values,keys} just made the migration so much more painful. I would say I spend a significant amount of time figuring out if I want to rely on six or if the array is small enough that I can live with unoptimal .items() in py2 throughout the migration. What would've been the issue of keeping .iter*() calls around, with a warning maybe? In general, there's a lot of stuff that could've been kept for backward compatibility, with a warning. It's much easier to slowly fix issues like that over time as you maintain code than to have to convert it all in one go, which put a giant barrier for people want to switch.
When are we going to start treating projects that are stuck on Python 2 the same way that we treated IE6 or Adobe Flash? I'm kind of sick of hearing the "well some people have to make technical tradeoffs in what they choose to support" argument. We're all trying to make better software here. Catch up.
I couldn’t care less about support for libraries and builtins. If my boss came up to me and said “Hey, I think we should use this Python 3.x library”, my response would be “Great, let’s use Python 3”.
That said, I understand there are certain applications that are not compatible with 3.x and the company does not have the resources to dedicate to rewriting it. So let’s suppose there is a valid reason someone is forced to use Python 2.7. In that case, the number one priority of this fork should be back porting the security fixes.
You can live without async, type-hinting, and f-strings. But please be responsible when it comes to security vulnerabilities.
It is possible that the transition from Python 2 to Python 3 is the worst-handled of its kind in the history of open-source so far. Is it any wonder, then, that things like this spring up in their wake?
Please, if you read this and decide it's a great idea, don't start bugging open source maintainers to officially support Tauthon in their projects. The most disruptive voices I've encountered in the last 6 months have been doing this.
Interesting. Is there anything here to deal with the three states[1] of "bytes" and "barray" vs "strings"? That was the hardest part, for me, to release an extension that was 2vs3 agnostic. Personally, resorted to some ugly hacks with version detection, barray detection, etc to make a 2/3 compatible extension.
[1] The three states being 2.x prior to recognizing bytes, 2.x sorta recognizing, and 3.x hard recognizing.
Is there a simple guide to understanding the str/unicode changes from python 2 to 3? I've made mistakes with strings that ruined every single piece of code I've upgraded from 2 to 3 unless I just littered the code with defensive six() calls ... and even then some things still broke.
I still don’t fully get why we can’t have ‘print x’ in Python 3. We would have avoided 10 years of schism for something that was soooo backward incompatible.
It's the people who are learning the language from scratch taking advantage of this simplification, because they don't need to learn the special syntax of print with all its warts (like trailing comma).
reply