Not to offend, but yeah, if you've already partitioned the work to be done, then of course it's not hard.
I'm pretty sure that anyone who's come into even vague contact with parallel programming understands that the hardest part is communication and synchronization.
Parallel programming is not hard; state management is hard. And for problems that are not embarrassingly parallel, unfortunately there is no silver bullet.
The author's article generally focuses on C (and possibly descendant languages), but the phrase I am critical of, does not. Furthermore, I explicitly consider a very broad selection of programming languages (many not C-derived) in my opinion. The author's phrasing, I'd argue, paints the entire concept of parallel programming as not hard.
There's some irony to the fact that you re-interpret my opinion as being very specific to C and (indirectly) posit that - in that specific case - parallel programming is hard, and then yourself go on to select a very specific case where parallel programming is not hard, because some matrix operations are independent.
I agree that there are languages that are explicitly built to make parallel programming easy. But in general, and not just related to c or c descendant languages, parallel programming is hard.
It sounds like a dumb tautology but parallel programming is easy for parallel problems. It is harder to say maintain several interlinked grid cells in a simulation in parallel than it is to partion a non-interacting set to N cores to run in parallel and let the main thread know when everything is done.
For decades, parallel programming was only done by people in the high performance world. It was (is) difficult and often painful. The languages, APIs and paradigms were broken in various ways. But they got the job done, and the relatively few people who actually needed to do it were able to.
It's only now that parallel programming is moving into the mainstream. Hence, it's only now that it's worthwhile for computing as a field and an industry to pour the resources into making parallel programming "easier." The benefit from doing it before was marginal.
I think this statement at the end of the article - 'There is a common myth in software development that parallel programming is hard.' - is misleading. Granted the author denotes explicit situations where it is not hard, but if it's applicable in general, then it is hard. Not a common myth.
Is parallel programming hard? Without any further details or specifics, yes it is. It is far harder to conceptualize code instructions executing simultaneously, than one-at-a-time in a sequential order.
Parallel programs really are not that hard to write. The hard part is figuring out how to make a problem into embarrassing parallel. Once that's figured out, the coding is simple. In this case, the language choice won't help.
Huh? It's a pretty widely accepted fact that correct & performant parallel programming is inherently a hard problem (barring invention
of a silver bullet...)
My point (and I think the points of others responding to you) is that parallel programming is not always hard. That's also what the author is saying.
The common myth - you're doing parallel programming? That sounds hard
It's not always hard. It really isn't! You don't need to be a genius or an expert to write parallel code.
Maybe where we're getting caught up is Cassie K's comment on ml engineering. You don't need to know how to build a microwave to use a microwave. In the same way, you don't need to be a genius or some deep expert in distributed systems to use abstractions that parallelize your programs
To write a parallel program does not require that you know what a mutex is. It just needs you to understand some simple algebraic (6-8th grade) properties about your functions (and, in fact, for library functions, they can be annotated as associative)
There is a broad spectrum of parallel programs. Somebody using a web server implementation? They've made a parallel application
Somebody running tensorflow or pytorch? Also parallel! Even for simple stuff!
You could be a beginner programmer and be taught to make parallel programs without understanding distributed systems. It's not always hard. It's not generally hard. The complex bits are hard. The simple bits use 8th grade math.
The point is that these easy cases may just referred to as "work to do". Using more than one CPU core at the time to process through a backlog of work is just...the normal case. Doing work simultanously is not what people refer to when they say "parallel programming is hard"; and the pedantery is in insisting that people word themselves to exclude trivially parallel work when what they want to talk about are the cases where parallelism is not trivial.
The conversation above is like this:
"Solving cubic equations is hard"
"Actually, solving x ^ 3 = 27 is easy, just take the cubic root.."
"Well that is not what we meant when we said cubic equations, we were interested in the general case..."
"I see equations like x^3 = y all the time, it is not that often you have a linear term too..."
Parallel programming is not hard per se, it's just that there's so much more to learn (compared to "sequential programming"): threads, processes, IPC, messaging, actors, queues, synchronization primitives (mutexes, semaphores, monitors, condition variables, critical sections, atomics... you name it) along with their gotchas, concurrent containers with their limitations, transactions, distributed computing with its performance implications, new design patterns (e.g. async/await, promises and futures). But hey, programming has never been thought of as easy, right? Yet, speaking of having it hard, it may be enlightening to compare software development with having to design a piece of (digital) hardware, where not only much has always been happening "in parallel" but one also must keep taking into account physical characteristics of various parts of the circuit.
I think a lot of people don't know how easy it can and should be to handle the easy cases. C++ has some parallel constructs, but they are not as simple as OpenMP because they're meant to handle the harder stuff.
I think it's important to show and use the easy ways before getting into the complex things.
I don't think he's saying that you shouldn't concern yourself at all with parallelism; only that you should focus on concurrency first and that will lead to easier parallelism. And he I think he is saying that decomposition and concurrency helps non-parallel programs stay simple and easy to understand. The benefits of concurrency are greater than just parallelism.
reply