You've seen the xkcd chart about automating and the cost/reward for varying numbers of repetitions per day? I think the concept that big businesses are behemoths stems from a fundamental understanding of what each type of business does each day.
As an arbitrary example, I think if we compare the number of employees onboarded vs the number of new applications shipped, small companies/startups and enterprises are polar opposites. My experience with startups is that you end up with a new web service for every 2-4 people (wholly anecdotal) you onboard as developers. For that kind of company, it makes sense to optimize shipping new web services; you do it a lot! For an Enterprise, you might onboard 100 developers just to maintain and develop existing systems. Writing new frameworks to make building apps easier doesn't make much sense; many of your developers are working on an existing system that would be expensive to port to a new framework. Meantime, your HR department is drowning trying to onboard 20 people a week.
Wasting money shipping this listing is fine; they did it once. This was released in 2013, and I wouldn't be terribly surprised to find out this, or some small bastardization of this, is still running as a production page for Prudence. The money they wasted, amortized over 6 years, is nothing compared to what I've seen startups waste in productivity while new hires wait for laptops, credentials, access, someone to explain some legacy crap that everyone had to deal with, etc.
You're painting with a broad brush a religious view of an idealized tech stack rather than the problem of motivations at different scales that aren't equal or necessary for each.
Large companies reward resume-grade impact, not cleaning up code, because there is literally no value added (that can be claimed) unless code is used or directly saves money. So spending expensive engineering time refactoring code for future hypothetical concerns isn't viewed as achieving anything, even if it sounds good. It's a shitty reality but necessary to collapse the often disconnect between software engineering and business.
Very large companies also have the staff to roll their own custom everything because COTS FOSS just can't scale up or out well enough at a reasonable cost.
By contrast, startups lack standards and working infrastructure, and are often forced to rely on force multipliers such as potentially expensive SaaS services where they own nothing, not even their data. The upside of startup startups is that there's nowhere for average or lazy engineers to hide: everyone must produce and do things in a maintainable manner or face the consequences of poor operations or paying the price of tech debt.
There are no free lunches, religions, or perfect solutions in real-world web ops. There is only minimizing classes of fires and being able to respond to them in a lasting manner in order to move forward.
meh. i work at a small shop. most of the code i see that sucks was written by competent programmers under high schedule pressure. the code is clearly not well understood anymore, but if you trace the history you can see it's due to a few years of tacking on features as quickly as possible. the technical debt surely slows us down sometimes, but hey that was a business decision, maybe even a correct one! core business functionality is rock solid, its mostly UI code that sucks.
I wonder what the average startup codebase looks like?
Shipping products is boring when the products themselves are boring. That’s why people want to work at small companies who have a lot more stakes in each releases and pay close attention to what they build, how it reaches market and make it evolves after.
To throw another log in this rant bone fire, a few decades ago software was seen as harder and more expensive, and while there was some frivolous and/or stupid apps, big enough development projects were usually bound to some purpose.
Our industry in its current form is more open to build meaningless things and trash after a while. I knew devs who were specialized in home page redesigns, and they viewed it as facelifts to show the client company is still alive. It has a business purpose, but in the grand scheme of things the only person that really enjoyed the project was the guy using it to learn vue.js.
Or a team porting an app from Objective-C to React native, not because they think it’s shiny but because hiring is hard/expensive and/or the business wants to have one size fits all apps for iOs and android.
The kinds of bills that some large company development environments end up running are surprisingly high, but those companies also think the opportunity cost of losing productivity on application team developers to also very high. That's how you find companies that decide to hire very expensive teams to, say, add types to a dynamic language, rewrite a standard library, invest in massive parallelism to run thousands of CPU hour integration tests in 15 minutes, or many other projects like that which no normal company would ever consider. If it gets some project team to cut delivery times by 3 weeks, it's all considered worthwhile.
Where we run into trouble is when we try to copy the shape of those big companies, and the kind of initiatives they fund, into startups, or mid-range companies. In quite a few ways, those top companies are better at development, but the base costs are only worthwhile if you have a money fountain of real revenue that grows far faster than your dev expenses. Trying to copy them with very different conditions is going to lead to tough problems, possibly company killing, and it's the kind of imitation we see all over the industry.
So yes, someone like Github is going to have eye popping expenses, because the alternatives to avoid those expenses just don't make sense to them. If you are not working in a multi billion dollar company, their practices can be interesting, but it makes as much sense to emulate them as it'd make for you to hire an entire Formula 1 pit crew to keep your commuter car in good shape.
There's definitely perfect and perfect for the business. But when large companies have many developers they all have to do something. They will spend their time doing something unnecessary.
If a piece of software is needed by the date of a major launch, and a programmer doesn't ship, all because he wants to reduce the amount of code involved, then he's an awful programmer as far as the business is concerned.
Running over time can cost millions of $ of investment in other areas of the business:
- Press organised
- Advertising purchased
- 1000s of hours of staff training underway
- Suppliers informed of increased demand / new supplies purchased
- Customers told to expect delivery by date
Any one of these areas can cost 10x or more of the cost of developing the code.
If you're not writing code which would be affected by that kind of criteria then cool, go for whatever metric you want to define.
Time taken is almost always the number one criteria of any business requirement, or to put it another way, small size of code means almost nothing to anyone except the original developers.
So... one of the places I work is one of those 'nimble little startups', and it gets more and more bogged-down by the day due to some poor technical decisions taken early. There are seven different kinds of timestamp in the database, for example. That's pure dev overhead that gets applied for everything. And because the nimble little startups are pressed to work on new features rather than refactoring, more tech debt just gets added to the pile...
Also, don't forget selection bias - most small companies, startups included, die in the first few years. Only the success stories make the papers.
I have no idea what you do (and what startup idea you tried before and got burned) but your generalization is not accurate. You can also say that developers are extremely particular about what they want to pay for. Developers spend hours to automate something that takes 5 minutes every week is because it's fun. Obviously my previous two statements don't apply to every developer, hence the generalization is wrong. Some are cheap, some are particular, and some are even more particular. Just like every other field I suppose, who would've thought!
I completely agree with this statement. Further, what scares me about this article is that he's basically reasserting every programmer's mindset in a different context: "Writing new code is awesome! Maintaining this giant mass of code sucks (regardless of if it's good or bad code)!" I don't have anything against the article or its ideas, but yeah, that maintaining a company of 86 of even the most self-sufficient employees is less fun than creating a web application you have complete control over is unsurprising.
There is also incredible bureaucracy, such that attempting iterative agile development is almost certainly doomed.
One reason many Web apps are so sleek and fit is that the developers could code, release, judge, and repeat daily.
This is much harder to do with large entities, who a) will need to present Accounting with the One True Final Cost for pre-approval and budgeting, and b) will need to have Official Sign-Off on the Absolute Final Specs from some dozen department heads.
The biggest (non-tech) companies seem to have the worst systems for developers. Writing code for a $500m ERP system for a $4b company? Here's a 19" LCD and a Pentium 4 2.4GHz and 3GB of RAM. You can only open one project at a time, blah blah.
Work at a 20-person company? Picked out the loaded MacBook Pro, monitor, and keyboard I wanted to use after my interview.
This is a valid point. I agree it doesn't seem like an efficient way of conducting a business in the long-term, but for a one-off problem that they happen to be stuck on, it makes more sense.
The feeling I got was that they weren't developers by trade (the code quality I mentioned was a red flag), very small, early-stage startup with no money and they needed expert help, fast.
This is more about Silicon Valley start-up culture than about how coding itself works in a perfect world.
Here's an example of how extreme it gets. Imagine you work in a commercial-facing start-up that blows up seemingly overnight.
In those kinds of companies, you'll have a core product that got super popular but was built by a small team. But now the investor money is coming in and everyone is taking about "throwing more coal in the engine". The company has to keep showing growth to keep investors happy but they essentially got lucky with the original idea and have no idea how to repeat it.
In an attempt to accelerate growth, management spins up 10s of new teams to build out new lines of business. They'll hire a bunch of product managers who will come up with lots of new spin-off products that need to be built ASAP so the CEO can talk about them with investors in the next quarterly call. All of a sudden the company has an ad-tech team, a customer loyalty tech team, an affiliate marketing team, a local retail commerce team, a physical gift cards team, a "whatever mobile SDK Apple announced at the last WWDC" team, and so on. And all these teams need supporting services so you get another wave of new teams who build supporting services. And of course now no one is really working on the original product that is generating the revenue because it won't get anyone promoted since it is already "mature". Everyone has moved on to new teams where they can "make a bigger impact" aka claim responsibility for moving a metric so they get more money.
Jump forward 24 months later and almost every one of those ideas will have failed to gain any real traction in the market and definitely won't have made a profit. But now a business that used to be run on a simple monolithic web app maintained by 20 people has been replaced by a 1000 person dev team building 75 microservices to power all these new verticals. But when all the new verticals fail to grow much, all the best developers move on in search of new projects more likely to get them promoted so you end up with a second wave of 2000 fresh college replacing them that all in aggregate generate barely any additional value over just having kept the original webapp going with the original team.
Could not disagree more. If you are a startup that is struggling to survive, it is an absolute waste of resources to overengineer your app for potential scale. It's the VC way to burn all of the money hiring talented infra engineers to manage this, but it is not pragmatic at all. The exception is if your startup is literally a high volume data processing platform or something similar.
That simple PHP/MySQL or rails monolith will scale much further than you think by throwing more servers at it without having to hire an army of devops engineers. Solving problems you don't currently have when your company is not profitable is a waste of money.
Yea sure some people in the sea of 500 will be doing elegant clean things, working in the core part of the business ( usually at or near the cash register program ) but if you have reason to have 500 developers you don't simply get the luxury of having every project be written perfectly.
Landing somewhere in-between by getting things done not perfect but shippable, and doing it fast as possible is par for the course ime. The best code is always written the 2nd or 3rd time, never the first.
Especially on the case of hyper growth worrying your competitors like lyft or postmates or amazon might get something pivotal out first.
People have to learn somewhere, usually things like this they learn at scale, on the job.
The SaaS shop that I worked in had two Java devs, two JavaScript devs, a python gluer, and a mobile developer.
Building the even a basic framework takes hundreds of hours of developer time... for no real competitive advantage.
It's fine to work on innovative things (there was a point were something about Java annotations clicked and the report writing part got rewritten in a week)... but building big things that give small gains aren't viable in small shops.
Doing innovation on "how can we compress this web page even more" makes no sense at a small shop - if it's an issue grab Minify off the shelf and use that. Don't spend half a year writing your own that you'll need to support.
The small company is best served by working on things that give the competitive advantage - not working on those big ideas (unless that big idea is the competitive advantage).
Big tech companies have the advantage that they've got thousands of developers some of whom have some free time or the competitive advantage is removing another 200 bytes from a web page.
That small tech shop had 4000 hours of Java dev time to "spend" a year.. and another 4000 hours of JavaScript, and 2000 hours of python, and 2000 hours of android/iOS. There's a stack of 5000 hours of bugs and features for each of those parts of the app. Spending 500 hours of anyones time on something that can be solved off the shelf for under 10 hours to integrate is far from an optimal use of time.
I agree with you but I think it's pretty straightforward how this sort of thing happens:
1. Startup builds thing fast in dynamic language because they need to optimize for development speed and iteration, not maintainability or scalability.
2. Startup grows and continues to hire for expertise in the tech stack they are mostly already using.
3. Repeat for some years and some hundreds of engineers and you arrive at this exact scenario.
As an arbitrary example, I think if we compare the number of employees onboarded vs the number of new applications shipped, small companies/startups and enterprises are polar opposites. My experience with startups is that you end up with a new web service for every 2-4 people (wholly anecdotal) you onboard as developers. For that kind of company, it makes sense to optimize shipping new web services; you do it a lot! For an Enterprise, you might onboard 100 developers just to maintain and develop existing systems. Writing new frameworks to make building apps easier doesn't make much sense; many of your developers are working on an existing system that would be expensive to port to a new framework. Meantime, your HR department is drowning trying to onboard 20 people a week.
Wasting money shipping this listing is fine; they did it once. This was released in 2013, and I wouldn't be terribly surprised to find out this, or some small bastardization of this, is still running as a production page for Prudence. The money they wasted, amortized over 6 years, is nothing compared to what I've seen startups waste in productivity while new hires wait for laptops, credentials, access, someone to explain some legacy crap that everyone had to deal with, etc.
reply