Yes! Had a startup client with a 4 person full-time "Big Data" team. They brought me into help them automate their server infrastructure. As part of the process I had them show me the Hadoop jobs they were running. 100% of the jobs could have been done as rudimentary SQL queries to their DB.
Welcome to the land of engineering, where we will fine-tune an algorithm to save milliseconds, yet waste man-years of engineering time to look like the big kids.
So many engineering choices are based on fashion rather than need.
Gotta get that hot new bag from Prada to show off to your engineering friends! :D
Yep. This is why I'm not sure that I want to do a deeper dive into algorithms. It seems like I could learn to implement all this awesome stuff with the least number of CPU cycles, but it's probably best to learn to communicate these things extremely well just to keep my sanity
To be fair, your employer is not concerned with your resume. They certainly don't pay you a premium for working on things that look boring or harmful on a resume.
If companies were serious about providing realistic advancement opportunities (in salary, in tech, in responsibility) there wouldn't be as much of an impulse to work on "cool" (i.e., marketable) projects.
Many dev jobs can be accomplished by pretty tried and true technology. For a startup and even most mid sized companies, MySQL and a web framework will get you pretty much everything you need. But any kid a year out of school can do that. So those positions are not really well paid, certainly not enough for a family in the Bay Area.
To become a mid-level engineer and make more money, you've got to implement some kind of complicated distributed system. Incorrectly. And then you can "fix" it (or someone else's) to get the next promotion!
This is stylized and not 100% true to reality, but I think it gets at a core truth.
There are many cases where there's a real need to create a solution either without any established framework or where previous attempts have failed. That means you're either a founding engineer or some skunk-works employee where you are given great autonomy but also bear most of the responsibility for the failed project. And, AFAIK, these positions are usually not advertised, you need good connections and luck to arrive at such opportunities.
The alternative is to realize that for certain types of problems, programming has become commoditized and one has to go up the consulting ladder selling architecture expertise, while improving social skills to be able to deal with customers at other level that isn't plain coding.
> Many dev jobs can be accomplished by pretty tried and true technology.
Then they should be.
> To become a mid-level engineer and make more money, you've got to implement some kind of complicated distributed system. Incorrectly. And then you can "fix" it (or someone else's) to get the next promotion!
Yea... that's dishonest, and frankly theft of the companies money.
Welcome to the land of engineering, where we will fine-tune an algorithm to save milliseconds, yet waste man-years of engineering time to look like the big kids.
So many engineering choices are based on fashion rather than need.
Gotta get that hot new bag from Prada to show off to your engineering friends! :D
reply