Again I'm not sure if I agree or disagree with this. My hatred of MPI is only outweighed by the fact that I can use it... and my code works.
I think a large part of the inertia behind MPI is legacy code. Often the most complex part of HPC scientific codes is the parallel portion and the abstractions required to perform them (halo decomposition etc). I can't imagine there are too many grad students out there who are eager to re-write a scientific code in a new language that is unproven and requires developing a skill set that is not yet useful in industry (who in industry has ever heard of Chapel or Spark??). Not to mention that re-writing legacy codes means you're delaying from getting results. Its just a terrible situation to be in.
I'm helping to organize a workshop about alternatives to MPI in HPC. You can see a sample of past programs at [1].
But you're right: today, MPI is dominant. I suspect this will change, if only because HPC and data center computing is converging, and the (much larger) data center side of things is very much not based on MPI (e.g., see TensorFlow). Personally, I find it reassuring that many of the dominant solutions from the data center side have more in common with the upstart HPC solutions, than they do with the HPC status quo. I'd like to think that at some point we'll be able to bring around the rest of the HPC community too.
First of all, this is from 2007, since then the number of openMPI deployments has increased steadily, the latest release is of the 4th of may of 2010, so to declare it dead is realy a bit premature, it looks like MPI is alive and kicking.
Appistry apparently makes its money by selling cloud management software. This is fine and good but does not overlap 100% with the main field of application for MPI, which is large scale number crunching in the scientific world.
This includes large scale simulations of all kinds, including fluids, particle systems, molecular and sub-molecular simulations.
While it is possible to run these on a different architecture by porting them, the amount of code that is written for and around MPI means that MPI is here to stay for a long long time. After all, rewriting all that stuff would take ages and cost a fortune.
Probably MPI will be around just a little longer than FORTRAN.
If only there was a standardized, robust, widely available cross-platform Message Passing Interface that could do this.
I don't grok why people outside of HPC seem to be shunning MPI. The shared-nothing memory model and asynchronous nature of MPI makes it very similar in spirit to a lot of the current web dev tech, AFAICT.
I was at a talk of his last year and there are a number of fault tolerant MP algorithms being drawn in. MPI hasn't been updated in ages, I don't think that necessarily means we need to ditch it, it just means the standard needs to be modernized. I don't feel very strongly about this since working with MPI is a huge pain in the ass and it seems like the challenge of modernizing it is just gargantuan.
Also I'm not familiar with spark, but isn't Chapel a decade old at this point and barely works at all? I tried their compiler last summer and it took 5 minutes to compile hello world, hopefully its improving.
Some of us have backgrounds in MPI, BSP and other distributed computing paradigms. I switched to high throughput computing and I can tell you that I would never go back to MPI. People who use MPI almost always use it as a crutch to avoid solving the hard problem that would make their computation a lot easier.
As an example I used to write MPI code for molecular dynamics simulators. The goal was to use a large supercomputer to scale the computation up to longer trajectories. But then, with cheap linux boxes it made more sense to just run many simulations in parallel, because low latency MPI class networks cost much more than the machines. We and others made the switch to other approaches which didn't need MPI- we ran many sims in parallel and did only loosely coupled exchanges of data. In fact the coupling was so loose we would run for weeks, dumping output in a large storage system and processing the data with MapReduce.
When I look back at the people who still do the MPI simulations- they are doing dinky stuff and can't even process the data they generate because they spend all their time scaling a code using MPI that didn't need to be scaled with MPI.
I'd love to see tools like MPI come back into mainstream. For instance, there's some work being done optimizing MPI in java for high performance computing but not nearly as much as what's being done for tools like Hadoop.
John: You make some solid points. Most though seem to more support the idea that more research investment/emphasis is badly needed for HPC programming models. From an application perspective, one sees such a dominant reliance on MPI+X primarily because typically the value proposition just isn't there yet for alternatives (at least in the areas where I work, where we have undertaken new developments recently and done fairly exhaustive evaluations of available options). Though the coding can be somewhat tedious and rigid, in the end these shortcomings have been outweighed by the typical practical considerations -- low risk, extremely portable, tight control over locality and thus performance, etc. It's obviously not all or nothing - as you say we could choose something even lower level and possibly get better performance, but when seen from the perspective of the near and mid-term application goals, it's hard to make a different choice unless explicitly tasked with doing so.
"Running MPI programs on a cluster of commodity hardware" was perhaps a sufficiently novel idea to warrant a unique name back in the 90s, but these days it's more or less the standard for all but the most exotic and expensive of scientific computing... and even those expensive custom supercomputers tend to still be x86 processors running MPI code.
Basically you don't hear about them any more because it's all Beowulf clusters now.
Too bad he didn't talk about GPGPU killing MPI too or not. I don't know enough to say.
I'm not familiar with the HPC space but I thought a lot of new work, at least in machine learning, was migrating to GPGPU instead of traditional CPUs. The compute per $ or per watt payoff is too large to ignore.
That's an excellent essay. I agree with about every point. I worked MPI over ten years ago. It seemed like an assembler language for cluster computing: something for raw efficiency or to hold us over until we get a real language. I was playing with Cilk and other parallel tools then. Much better but little adoption or optimization.
The examples given in Spark and Chapel show how much better it can be. Such methods deserve much more investment. The bad thing is that the DOE labs are clinging so hard to their existing tools when they're in the best position to invest in new ones via huge grants they have. They built supercomputing before anyone heard of the cloud. Their wisdom combined with today's datacenter engineers could result in anything from great leaps in efficiency to something revolutionary. Yet, they act elitist and value backward compatibility over everything else. Starting to look like those mainframe programmers squeezing every ounce of productivity they can out of COBOL.
I think the change will have to come from outside. A group that works both sides with funding to do new projects. They either (a) improve MPI usage for new workloads or (b) get the alternatives up to MPI performance for HPC workloads. Then, they open the tool up to all kinds of new projects in both HPC and cloud-centered research (esp cloud compute clusters). Maybe the two sides will accidentally start sharing with each other when contributing to the same projects. Not much hope past that.
reply