I think my experience in async in other languages has shown how rough async in Python is.
Unless async is the default python, it will always be an after thought and introduce needless complexity when working on large projects that require a lot of external libs.
python's async mostly works as long as you're using async-native libraries. if you can await it, it's okay. it only gets disastrous when you have to start mangling callback code into promises.
although i really do love javascript's ability to await synchronous functions without it causing any errors. i'm sure it's less performant, but sometimes i just don't care.
The thing about Python is that unlike most other languages, you don't have to deal with that complexity. You can still write Python 2.7 style code, no problem.
This approach to async, though, is just a language feature that's becoming mainstream right now. C# has it, ES7 has it, C++ has a working paper on it etc. Python actually had the benefit of watching how things work out elsewhere before implementing it all.
Async in Python is a huge deal as it is in any language, yes. For very real reasons (cutting down on incredible amounts of confusing boilerplate) to very lame reasons (it's been memed into developer consciousness enough that it becomes a primary yes/no gate for development teams).
I think the forcing of Async into Python is largely a waste of effort. There's better languages and tooling for those jobs. Effective devs use the right tool for the right job. I've had to work on an async Python codebase the other day, and was surprised at how janky and cumbersome the syntax is (though I shouldn't be surprised). The whole setting up the event_loop, create_task, run_until_complete, etc. just feels pretty gross. I'd rather just implement something I knew to be performant in something like Go, C#, etc. a language that supports first class coroutine-style/message passsing support...
Slightly off-topic: I find async in python awful. The idea and concept is great, but the implementation of coloured functions in python is terrible. It remembers me of the "transition" from lists to iterators (mainly python2 to 3). Coloured function support should drastically improve (i.e. calling an async from non-async) in order to be accessible for newbies.
Honestly IMO Python does a good job these days without engineers having to reinvent the wheel. I've used asyncio with Starlette and/or FastAPI in larger-volume systems without much issues. I'm curious what specific issues the author had with async Python, and when that was since I don't see any such issues in my experience.
I've used python since 1995 and I can say that async is one of the worst things I've seen put into python since then. I've used a wide range of frameworks (twisted, gevent, etc) as well as threads and even if async is a good solution (I don't think it is) it broke awscli for quite some time (through aiobotocore and related package dependencies). It's too late in the game for long-term breaks like that or any backward-incompatible changes impacting users.
Coming from the JS world, asyncio feels fine to me. I've noticed a lot of Python programmers who have a strong background in C and Python struggle with asyncio. But in their case, I think it's more them disliking the async programming paradigm rather than the specific asyncio API.
Might be unpopular but still good to stick to it. I tend to associate excessive enthusiasm for pythons async await framework as a sign of immaturity and potential for doing things without understanding them fully. I’d never trust async code from anyone except the best python engineers. Most of them out there can’t even debug linear code, adding async to the mix only makes it worse. At least you can trust the time stamps.
There are too misconceptions about async python I keep seing again and again:
- async python is faster, but for a very specific niche of workload. Most tasks you do don't fit that workload at all. Which mean most of the time, you should NOT use async Python. And it's ok. Don't make your code complicated when you don't need it.
- async python is not just for performances. It helps with making some specific kind of concurrency easy to reason about, because the context switch is explicit, and the chain of event can appear linearly in the code, thanks to await.
- it's ok to have some async processes and some async processes. It's not one or the other.
- async is not a replacement for threads or processes. In fact, there are good use cases for having several processes, each with several threads, each with one event loop. Which means that if you benchmark your WSGI lib with 16 workers, you should do so as well with your aWSGI one.
The corollary to this is that some website loads are very well suited for async (e.g: an SPA with a lot of connectivity which delegates long running code to other services), but a lot are not. If each request make a long SQL query, seek the hard drive, dynamically preform i18n and add some calculation on top, it may very well be better to use sync for this.
So when would you use async python ?
- For performances, when, you need to maintain numerous long connections. E.G: doing websockets ? Use asyncio. Serving static files without nginx ? Use asyncio. Want to create a web crawler and your memory budget to allowed to open 10000 threads ? Use asyncio.
- For the interface, use the async/await keywords everywhere you need need inversion of control for I/O. It's not just about perf here, It's also a mechanism to delegate arbitrary parts of your code with a common standard interface around a fancy state machine. You can use that to abstract all your I/O and switch backend at will.
I love Python async - it’s a complete game changer for certain types of applications.
I find Python async to be fun and exciting and interesting and powerful.
BUT it is a big power tool and there’s so much in it that it’s hard to work out how to drive it right.
I have pretty good experience with Python and javascript.
I prefer Python to javascript when writing async code.
Specific example I spent hours trying to drive some processes via stdin/stout/stderr with javascript and it kept failing for reasons I couldn’t determine.
Switched to Python async and it just worked.
The most frustrating thing about async Python is that it has been improving greatly. That means that it’s not obvious what “the right way” is, ie using the latest techniques. This is actually a really big problem for async Python. I’m fairly competent with it, but still have to spend ages working out if I’m doing it “the right way/the latest way”.
The Python project really owes it to its users to have a short cookbook that shows the easiest, most modern recommended way to do common tasks. Somehow this cookbook must give the reader instant 100% confidence that they are reading the very latest official recommendations and thinking on simple asyncio techniques.
Without such a “latest and greatest techniques of async Python cookbook” it’s too easy to get lost in years of refinement and improvement and lower and higher level techniques.
The Python project should address this, it’s a major ease of use problem.
Ironically, pythons years of async refinement mean there’s many many many ways to get the same things done, conflicting with pythons “one right way to do it” philosophy.
It can be solved with documentation that drives people to the simplest most modern approaches.
To be fair, Python asyncio is a really ugly interface for something that other languages (e.g. JS and others) have done a lot better. Maybe look at other languages than Python for async programming.
The asyncio module was heavily refactored in 3.7, which introduced the async/await keywords which was a *huge* improvement over the 3.6 and below implementation.
I don't agree with this, but I will concede that async Python is rarely the correct choice.
Unless async is the default python, it will always be an after thought and introduce needless complexity when working on large projects that require a lot of external libs.
reply