Most episodes of Software Engineering Daily are interviews with an expert about a technical software concept. Occasionally I write editorials, and also record them as a podcast.
The first editorial was about 10 Philosophies for Engineers, the second was about how poker relates to software engineering, and the third was about music and software engineering.
Today’s episode is called “You Are Not A Commodity”.
Since it is a departure from the normal format, I am releasing it on a weekend, and there are no ads on this episode. If you don’t like this episode — don’t worry. On Monday, Software Engineering Daily will be back to the usual highly technical rigor. In any case — please send your feedback on Twitter or email.
You Are Not A Commodity
Software engineers are often trapped in careers that consist entirely of software maintenance rather than building new products.
This post:
During the industrial revolution people began working on assembly lines.
At the assembly line, workers were given narrow roles. The assembly line is a resilient, consistent tool for manufacturing because the roles of each worker are so well defined that the worker can be easily replaced.
Assembly line workers are so easily replaced as to effectively be a commodity — a raw good to be bought and used up.
For the worker, becoming a commodity was unpleasant. But there were not good alternatives to working on an assembly line. You could not simply quit the assembly line and go start your own.
Individual workers suffered, but society progressed quickly because industrialists could leverage the commodity workers to have more predictable output.
This method of production continued into the the information age.
Early computer systems were hard to operate. Individuals did not have much leverage. Teams of people needed to be closely managed to accomplish significant tasks.
Today, computer systems are easier to operate. Individual engineers can build lifestyle businesses. Small teams can build products as big as WhatsApp. Everyone has more leverage — not just industrialists who are in charge of assembly lines.
Most software engineers take a career path that gives them as much leverage as the commodity workers of the assembly line. We conform ourselves to the unpleasant tasks that are needed by large corporations.
Why should I care? Is this bad?
If you take one thing away from this post, make it the following: it is now easy to build your own assembly line.
The economics of the big company do not reward the lower level workers. Today, it is easy for a skilled engineer to opt out of the poorly compensating corporate machinery.
20 years ago, the tools within a large software corporation were much more sophisticated than anything you could buy as an individual.
Then Amazon Web Services reduced the cost of server provisioning and management from $50,000 for a machine to essentially free.
The excellent free infrastructure provided by AWS led to excellent free software like Dropbox, Trello, and Slack. These cheap tools integrate with each other and empower the individual to be massively productive.
SaaS tools get combined with each other and built into even better tools. The quality of available software compounds at a staggering rate.
Cloud computing triggered such a massive improvement in public software that individuals and small teams can build a tech stack that often works more effectively than that of a big corporation.
Big corporations are tightly coupled to their tools, and cannot take advantage of this rapidly compounding software quality. Outside of a big corporation, you are free to mix and match tools as you see fit.
Software tools are the means of production, the equivalent of the assembly lines of 1900s industrialists, and access to these means of production have become decentralized.
In fact, the entire economy has become decentralized.
Not just means of production but distribution, contracting, payments, access to domain expertise, and every other aspect of business. The advantages of big centralized corporations are now available to individuals and small teams.
Fiverr is one example of a service provider that an individual or small team can use for leverage. Fiverr contractors can provide cheap help with accounting, WordPress, audio and video editing, marketing, and other supplementary work that big companies hire full-time employees to do.
A large corporation needs recruiters and hiring managers employed to vet potential employees. Instead of the time-consuming interview process, Fiverr’s reputation system means that the work of candidates speaks for itself. Interviewing is a formality.
Fiverr is useful for tasks that are not complicated, but require focus and consistency. Upwork, a similar service, can be used for more complex tasks, like software development. These services fulfill the dream that was believed about offshoring software development in the 1990’s — high quality, cheap outsourcing.
Fiverr and Upwork put downward pressure on the cost of contractors, but software businesses also need full-time workers. Building a full-time engineering workforce has also become cheaper.
This sounds counterintuitive to anyone who has heard of the “peak talent war” of tech, but each individual engineer has so much leverage today that you need dramatically fewer engineers to ship good software. You also don’t need to pay for real estate to house those engineers.
In the past, a team of software engineers needed a physical place to work together in. That is no longer the case. Docker, TopTal, and Hashicorp are among the companies that have moved to remote work, enabled by Slack and other collaboration tools.
Remote companies report increased transparency and improved communication. Remote employees must leave a trail of finished work to let themselves be audited — which is not difficult for employees who are actually getting work done. Chat logs and github commits do not lie.
At a large company, engineering productivity is reduced by having to interface with project managers and business development people.
Small companies benefit from the fact that engineers have to understand the big picture — business, design, sales, and product development. Engineering manager Fred George’s concept of Programmer Anarchy takes this to the extreme.
The success of Programmer Anarchy indicates higher leverage among engineers. The role of engineer can subsume all other roles.
Programmer Anarchy works today because the tools are better and the practice of software engineering is more widely understood. Engineers don’t have to spend as much time writing boilerplate code, so the leftover time can be spent thinking about the product at a higher level — which yields business value.
Understand this: the leverage of the individual engineer is big — much bigger than it has ever been. The leverage of the small team is massive.
If you are building a business alone or with a small team, the classic advice is to “make something people want” or “live in the future and build what seems interesting”.
This is good advice for the brainstorming phase of building products, but many engineers have never built a prototype. Engineers who have done maintenance for their whole career can’t imagine how to go from an idea to a usable product.
If you have spent most of your career at a large corporation, you might be handicapped by a reliance on internal tooling and workflows. If you want to build a product end-to-end, you have to devote some time to getting decent at every area of the stack.
You have to learn to prototype, which means building new programs from nothing. This is more of an emotional and psychological journey than a technical challenge.
We all remember the pleasure of building software from scratch, whether it was a simple game or a calculator app. Why do we ever stop building new stuff? Because our corporate software jobs normalize the idea of software maintenance.
If you have just escaped a corporate job, you can start with 1–2 months of re-skilling. You probably have some money saved up, and it might even be worth going through a coding boot camp.
Boot camps like Hack Reactor are renowned for their ability to instill engineers with the skills and the necessary workflows to build projects alone.
Even if you have a computer science degree, there is no shame in going to a coding boot camp. Universities teach theory, boot camps teach execution.
Whether you attend a boot camp or lock yourself in your apartment with a Pluralsight subscription, the re-skilling process can iron out the sad past life of the software maintenance engineer, allowing for a rediscovery of the creative joy that probably made you fall in love with programming in the first place.
This means getting away from the thick, industrial languages like Java and C#, and going deep on a set of tools that allow for rapid prototyping, like React and NodeJS or Ruby on Rails.
An engineer who is interested in data science and machine learning can spend time learning TensorFlow or Spark — there are countless tutorials online.
Building an entire prototype and deploying it to Digital Ocean or Heroku is a satisfying experience — especially after spending several years on the software assembly line, understanding only small parts of an application.
The first time you type in the URL of a webapp that you built and deployed entirely yourself will provide a rush that no six-figure programming job can provide.
With enough practice, building your own software becomes easy. The challenge then becomes what to build.
After acquiring fresh skills and learning to prototype, it is easier to dream up new product ideas without the self-doubt that you won’t be able to bring such a product to market.
Once you have the requisite skills to build a software product, what do you build?
Much has been written about the brainstorming process. Paul Graham essays and books like The Lean Startup are useful guides to thinking up an idea for software that generates money.
After spending a long time in a large corporation, your ability to think up fresh ideas might have become weak. As a programmer trying to think up new ideas, it is easy to become discouraged and think that everything has been invented already.
One strategy for overcoming this mental atrophy is to look at the giant markets that would be too big to capture even if all of the programmers in the world pursued them at once.
“Machine learning + X” is one of the safest routes to take if you are building a software business and don’t know where to begin. As futurist Kevin Kelly says, “The business plans of the next 10,000 startups are easy to forecast: Take X and add AI.”
Every business needs machine learning applications in the same way that every business needed a web site in the 90’s. At the same time, most developers are afraid of machine learning because it sounds intimidating.
When developers try out the available machine learning tools, they find scikit-learn to be as accommodating as Ruby on Rails is for web development.
Even if you build a machine learning product that nobody needs, you will have to learn machine learning in order to do that. If your first product fails, you might be able to reuse some of the code in your next machine learning product.
Hardware is another field with wide opportunity. Like machine learning, hardware also sounds harder than it is (“hard” is even in the name).
Just as Spark and scikit-learn are simplifying machine learning, cheap hardware like Tessel and friendly software frameworks like Johnny-Five are simplifying hardware prototyping.
IoT is the place to start for new hardware developers. Every major cloud provider and chip producer is investing heavily in IoT. Since giant companies are competing to provide the best IoT platform, individual developers can take advantage of that competition.
For the developer, everything is cheap. Companies like GE, Amazon, and Microsoft are fighting to capture market share, and they are willing to lose plenty of money bringing developers onto their platform.
Common customers for IoT products are factories, farms, warehouses, and other industrial businesses. These sectors also have huge piles of money to spend, and they are becoming more tech-savvy and willing to experiment with new hardware.
Regardless of what product you build, every enterprising software engineer should keep in mind one fundamental trend: the biggest pair of economic forces working in your favor are cloud computing on the supply side and small devices on the demand side.
Small devices like mobile phones and IoT need new, unique applications. Customers will pay money for these. Cloud computing means that providing those applications to the end user is cheap.
This trend will impact whatever area of technology you decide to work on in some way.
In developing countries, the demand for unique smartphone apps is even more acute than it is in the developed world because consumers in developing countries often do not have a laptop or desktop computer. The only computer they have is a smart phone.
Today, many developing countries have poor connectivity, which makes the demands of consumers different from developed countries. These places are less saturated with technologists, so they are ripe for people looking to study the local markets and understand the pain points of consumers.
There are plenty of other burgeoning sectors — the gig economy, virtual reality, Docker, cryptocurrencies. After studying any one of these sectors in detail, it becomes clear how much technological growth can be realized in the next few years, and how few people are putting themselves in a position to capture it.
The hardest battle to fight here is understanding that you should build something. Once you internalize that fact and acquire the necessary skills to prototype, the process of building is a fun exploration, even if you fail along the way.
Now that we have discussed how to build an assembly line, let’s address some counterarguments that favor being a commodity employee of a large corporation.
The compensation counterargument: corporations pay lots of money
The average employee realizes a small fraction of the value created by her work.
Most of the money goes to the people who created the assembly line, not the people who maintain it and build on top of it. Clearly, individuals who create an assembly line are rewarded more handsomely.
The loyalty counterargument: someone has to do this work!
Employees often feel a sense of guilt when they leave a large company. Their work is so important to the company’s health — how will the company run without it?
If your work is so important to the company, that probably means you could go to your manager right now and ask for the biggest raise that is justifiable given the revenue generated by your work. If they don’t give you the raise, that means they won’t have trouble replacing you.
If the company does give you a raise, that implies that the company could have calculated that your work was more important than your compensation implied — in which case the company was deliberately paying you less than it could afford. This doesn’t sound like a relationship where “loyalty” should play a major role in our career calculus.
Your relationship to a big company is firstly transactional. The company needs to pay you the minimum and extract the maximum. There is room for feelings of loyalty only at the margins of this relationship.
The stability counterargument: large corporations are safe
Corporations present a career ladder that can appeal to a student who progressed comfortably through the linear path of high school and college, where success was simple to plot.
This straight path seems quite comfortable. Do the work you are assigned and you will naturally rise through the ranks. In the average case, life is good!
The average case always looks good for strong engineer. If you are a good engineer, your downside risk is capped in most conceivable futures.
It is almost impossible to imagine a world in which engineers are not in high demand. In the hypothetical scenario where a good engineer is not employable, it is likely that something will have gone drastically wrong with the economy.
This fact negates the idea that patiently saving for the future is “safe”. Which financial instrument could you keep your money in that would insulate that money from a total economic collapse? Even if you have $200,000 in guns, gold, and bitcoin, will you even have the self-sufficiency to know how to properly manage those assets in a time of economic collapse?
Assuming the world economy continues the way it is going, the path of the corporate ladder is much riskier than starting something on your own. If you start something on your own and fail, you can always return to a large corporation, probably with improved skills.
The safer investment is to train yourself with the ability to invent and be independent.
The leverage counterargument: corporate infrastructure empowers me
Innovations made by internal employees
Google, Amazon, and Facebook retain their best employees by giving them freedom to invent. The employees who built AWS, or the Like button, or the HoloLens escaped their commodity status by building something that differentiated them so much as to become a lynchpin within the company.
Inventing from within a large organization is the perfect strategy for someone who is stuck at a company on an H1-B, or who is working off student load debt, or who has a family situation that prevents them from quitting their job.
If you aren’t restricted by immigration constraints or financial constraints, you have much more to gain by building something on your own outside of a company. It raises the stakes and forces you to overcome your fears.
When you invent a product on your own, the best case scenario is quite appealing. You get to capture 100% of the product’s value and you maintain complete creative control. Neither of these are guaranteed when inventing at a big company.
The mentorship counterargument: a corporation is a great place to learn
Corporations can be a great place to learn how to do highly structured work. You are assigned tasks, you learn how to use tools like git, you learn how to sit through meetings and talk to co-workers.
Reviewing code from other co-workers can be educational, and teach you how to improve the efficiency and readability of your own code. The corporation will impose reasonably high standards on your work.
The problem with “mentorship” at a big corporation is that it brainwashes you in the same ways that it has brainwashed the people who will be your mentors.
As an inexperienced employee, your work is likely to be centered around software maintenance. The company will pitch it to you as fresh and exciting because they see you as naive. But you won’t be building a new product where you have creative control and true responsibility.
The hardest part of building a new product is deciding that you are going to build a new product, and following through with that decision. Corporate mentorship is not going to focus on this.
If your long-term goal is to build new products, avoid the farce of corporate mentorship. If your goal is a career of software maintenance, then commence the brainwashing.
Building new software is the perfect activity for someone who likes art, science, business, or adrenaline.
Other activities that provide a similar emotional high include drugs, video games, sex, gambling, browsing Facebook, writing music, painting a picture, international diplomacy, raising children, and skydiving.
Humans get habituated to most of these activities and have to constantly raise the stakes. We need to gamble for more money. We need more intense drugs and more immersive video games.
When we build software companies, the stakes automatically get raised over time. As users grow, our financial swings get bigger. Users who love the products we build give us praise that is more meaningful than the Likes and comments on Facebook. And the intellectual challenge is constantly morphing and presenting new problems to solve.
Inventing new software is sustainable, rewarding, and fun. It is an adventure!
Many engineers work a job where they feel like a replaceable commodity. They spend several hours a day pondering their existence and doubting their self-worth. Escape this trap — build something!
