As an experienced Python dev... Python's "convenience" and "soeed of iteration" completely falls apart when you have more than half a dozen people working on the same codebase of varying development experience. You spend so much time digging yourself out of abuses of internal APIs.
Very much agree. I oversee a relatively small python codebase, but getting good quality, safe code out of the developers in a controlled way is really hard - there are so many ways in which the language just doesn't have enough power to serve the needs of more complex apps. We have massive amounts of linting, type hinting, code reviews spotting obvious errors that would be just invalid code in other languages.
It's like getting on a roller coaster without a seat belt or a guard rail. It's fun at first, and you will make it around the first few bends OK ... then get ready ...
Of course, with enormous discipline, skill and effort you can overcome all this. But it just leaves the question - really, is this the best tool for the job in the end? Especially when you are paying for it with horrifically bad performance and other limitations.
Python doesn’t scale well beyond one file, one developer, one time write. Once a program needs maintaining, updating, expanding, debugging - the experience quickly deteriorates.
I do love Python, but I often wish there was more rigor for some things. I currently work on a large project (>500K LOC) and have started to see it fall apart with a bigger team. Lack of enforced typing in function arguments, inability to create strict interfaces, inconsistency in standard library conventions, and package management would probably be my biggest gripes.
Are you the only person working on it? How big are we talking here, because I’ve worked on a few Python projects that fall in the region of “a couple to a few thousand lines” both by myself and with a team. Working by myself was significantly easier but Python definitely hits a “tangled mess” point, regardless of how disciplined I was.
In comparison one of my personal projects is in Rust and while that’s currently maybe only a thousand lines it’s already significantly easier to deal with.
Biggest problem I had with a largish Python codebase was code navigation and knowing the structure of the data going into and out of methods/functions. I had to attach debuggers just to see what was going on more often than I think should be necessary in a professional setting.
While this is something I very much appreciate about Python I think it's also true that structure needs to increase with the size of your codebase and the size of your team to avoid being driven mad. Python takes the "just apply discipline" approach which can absolutely work for lots of code but falls apart a bit with large heterogeneous teams.
In a relatively terse language like Python, anything beyond a few screenfuls of code is already "large scale" development. It's unwise to keep it all in a single module.
Every time we have to have someone fix one of our relatively few python projects I rue the day I wrote them in python -- it often takes hours just to get it so the code runs on their machine.
No programming language design survives first contact with the programmers. This is also holds on project level if you are 100 persons working on the same project it's very easy to mess it up.
I've seen lots of bad python code, I don't think it's hard to do the wrong thing.
No enforced static typing and no proper debugger make Python painful for large code bases. It’s good for scripts, prototyping, and gluing libraries together to make utilities, but if something expands beyond a single file I stop wanting to use Python. Convincing my employer of this is another matter and why I would rather avoid it completely.
In my experience this is quite typical in the python community, I think it's a consequence of the batteries-included philosophy. Many people are very averse to using primitive data structures and operations to solve their problems, so working with a python codebase is like gluing libraries together
Like many people here, I've worked on large python deploys for most of my professional career and have, over time, come to see python's downsides more clearly.
I also wonder, thinking back over my experiences and reading others here, if we are mis-ordering how these things come to be. Maybe it's not that python codebases are, for a given line count, worse than other languages. Maybe it's that python allows teams to continue using practices that aren't suited to their current scale longer than other languages. The reason that we all have seen hulking, monstrous, nearly unworkable python code bases is that most other languages would have already collapsed under the mismatch between approach and desired outcome.
I think we often approach software engineering as a puzzle - where you have a set of inputs (a too-large codebase, for instance) and a question of how to a better state. But programming projects are path-dependent creatures. Huge codebases must develop over time - they don't spring out of the heads of developers fully formed. If you were a typed language and your engineers had a lower LOC output per-day, then of course your code base will be smaller. Is that better? You iterate more slowly, but the scale of your code is also more manageable. In my experience, the challenge of large python code bases comes from the context that you need to understand from the surrounding environment: what are all these objects, what are their objects, etc. Typed languages force you to carry around more context, so any given function is easier to read, but you can still write un-navigatable code.
Couldn't agree more, but what I don't understand is that Python is plagued by this problem and nobody seems to have any issue with it. Trying to understand large Python codebases drives me crazy.
Most of python system i've seen is full of bugs, lack of stability, which causes huge damage from developer resources as well as operational costs. It's too hard for a junior Python developer to produce a reliable system.
One of the reason as i know is, they have no idea of using unit test.
Granted, I'm just a scientific programmer, but my workplace has a full blown software team maintaining a multi million line codebase. That codebase is rebuilt every night, and as I understand it, you're not allowed to submit a change that breaks anything. And they have people whose job is to keep their tools working.
What people casually think of as "Python" is really a huge dynamic ecosystem of packages. Imagine that there are 60k packages, each with 1k lines of code... that's a 60 million line codebase, and it can't be checked for breaking changes. Short of continually testing your code against the latest versions of packages, you're going to hit some bumps if you haul an old code out of the vault and try to fire it up on a new system.
I don't know how Javascript developers handle this.
I handle it by running Python inside an isolated environment -- WinPython does this for me -- and occasionally having to fix something if a new version of a package causes a breaking change.
The drawback of my method is deployment -- there is no small nugget of code that I can confidently share with someone. They have to install their own environment for running my stuff, or take their chances, which usually ends badly.
I've dealt with python noob codebase that had many deep side effects over multiple (~2000loc total) files. The overall logic fit into a single page, bugproof and easy to augment using FP idioms.
reply