How to avoid over-engineering

Every professional, especially a programmer, wants to write the code in the right way, in accordance with accepted standards and good practices. We want to do our job as best as possible: without errors, without omitting any condition, and considering all possible use cases. Such approach seems completely natural and desirable, but may actually lead to many problems in the project. What if our excessive diligence and perfectionism when writing code outweighs the needs and requirements of the client, his budget and agreed deadline?

In this article, I will present what is characteristic of over-engineering, its causes, ways to avoid it and deliver projects on time, without compromising the quality and satisfaction of both parties: the client and the developer.

Table of content

• Definition
• What’s the problem?
• Human factor
• Symptoms
• Causes
• How to avoid over-engineering?
• Over-engineering vs business
• Summary

Definition

Fagner Brack in his article provides a spot-on definition of over-engineering (https://hackernoon.com/how-to-accept-over-engineering-for-what-it-really-is-6fca9a919263#.da44lidym):

“Code that solves problems you don’t have.”

I am tempted to say that this definition of over-engineering can be applied not only to the code but also to system architecture, tools used, ways to carry out broadly understood processes in a project, company or organization.

Every occupation related to production is susceptible to this basic trend in which one loses sight of real and immediate problems as a result of being too obsessed with the concept. However, the problem with this attitude in the IT industry is that software development does not work in the same way as creating most other products that we use every day.

What’s the problem?

Creating products usually follows a specific project (concept) created earlier, and the whole process is based on a tried-and tested methodology that ultimately results in the production of the product. The product is reproducible, i.e. every copy coming out of the production line looks and has the same properties. The same amount of material was used for its production, the cost of production was the same and the whole process took the same amount of time.

In programming, despite various established processes and rules in place, two identical products can be made in a completely different way. Since there is no single scheme to write code for specific functionality, the character and size of the solution vastly depends on the contractor. As IT specialists, we tend to excessively develop business solutions. We like to blow our ideas out of proportion. Instead of simplicity, we favor a complex implementations, take into account conditions that don’t necessarily result from current business requirements. We consider ourselves smarter than business, and often make arbitrary decisions. We commit to deliver features at a later date, often exceeding the previously set budget but with the assurance of excellent “quality” and the benefit of the project.

Many programmers forget about their raison d’être – meeting the client’s needs, automating complex operations and simplifying, not complicating the client’s life. Of course, we are specialists and must make every effort to avoid technical deadlocks, cost explosions etc. We are not here to create the architecture and solutions of our dreams at every opportunity.

Context and human factor

Before starting a new project, you must first learn its scope and context of use, and only implement the project on their basis. Fagner emphasizes in his article that what constitutes excessive engineering depends not only on the technological decisions of people but also on the context, or rather the lack of knowledge of it.

If the domain is not exactly known, there is a great risk that the assumptions about the implementation will be incompatible with the actual requirements and needs. We then try to anticipate requirements and potential future use cases.

It is understandable that sometimes requirements cannot be made fully available at an early stage of the project, but maximum knowledge and use of them is necessary for the project to succeed and that we can avoid adding huge functionality in the form of patches. This scenario will result in software that is very difficult to maintain and scale.

On the other hand, as programmers, often despite knowing the context and requirements, we like to make assumptions about them anyway. It often seems to us that we know better than the customer what the project needs. Our knowledge and experience often pushes us into redundancy, complex techniques and forced design patterns. All this is supposed to protect us from something that may never come. We set high walls and cannons while our actual enemy is the proverbial fly.

As parts of our architecture, we tend to incorporate new trends that may not be good for our project, often without verifying the benefits of implementing these practices first. We fall prey to technological hype, we follow specific design patterns only because they are known and liked. We could achieve the same effect in a much simpler and faster way without a great detriment to quality, and yet we focus on greater complexity and apparent professionalism of our solution.

We often desire generic solutions hoping to apply them for a larger pool of cases. As a result, instead of reaping the benefits of such approach, we run into ifology and unnecessary complexity of our code. We introduce numerous – so liked by programmers – abstractions, leading to enormous complexity and lack of readability. As a result, a new person or the author himself will not be able to understand his own code in a few days or weeks.

The most difficult question is – how do you know that we’re dealing with over-engineering? What is the border between the right and the recombined solution? When should we take action, what symptoms should we look for and how to deal with such a situation?

Symptoms of excessive engineering

We can say about the occurrence of over engineering when:

• you write code that solves problems you don’t have
• you consciously want to be technologically ahead of others and make it show in the code at all costs
• you have completed the implementation of a specific issue in accordance with business requirements and finally you start to think about the boundary conditions you invented. You are just a step away from over-engining
• you decide to ignore the rules in the team / project / company because you believe you can do something better and you know the business requirements of the project better
• there is a gap between the complexity of the problem and the consequences of the solution
• you try to prioritize tasks, but in the end you pay too much attention to tasks that are not currently of the greatest business value
• you are trying to reinvent the wheel. Nowadays, there are so many libraries and ready solutions that it is almost certain to find the answer to your problem
• you use wrappers and adapters for everything. Wrapping ready solutions with your code is useful, but it’s easy to overdo it
• you are guided by deeply-rooted “beliefs”. It’s the most important, but also the most difficult thing to realize. Look at your code and see which parts seem too complicated. Ask yourself if there is an easier way to implement each of them and think about the reasons why didn’t do it. If you do not have a good answer, this part of your solution is probably over-engineered
• Other signs of over-engineering may include over-creating of abstract, reusable mechanisms, over-applying DRY, and failing to comply with KISS and YAGNI principles.

Causes

What exactly contributes to over engineering?

• willingness to make your solution / project scalable, universal and at the same time compliant with the latest standards, principles and trends
• boredom at work. When you have too much time, you can spend too much time excessively polishing your code
• writing everything from scratch. There is a huge chance that someone has already done it. What’s more, he did it 1000 times better than you, because the community has already tested this solution, report bugs that were then corrected. Chances are this solution is already recognized as an industry standard
• problems with setting priorities,
• the code mostly deals with the implementation of non-business technical problems, business issues come second
• ignoring good design principles or SOLID principles. If you think that you have written the perfect code, it probably spells trouble, because extremely rarely someone writes something that cannot be written better
• programmers often think that they are the smartest people because they write the code to create projects. Although we can predict 100 different cases, business will come up with the 101st we never thought of. We simply assume that we have everything under control, but don’t fully realize what we are dealing with
• we try to group and generalize logic as much as possible. Business requirements usually differ, rather than overlap
• sometimes duplication is better than generating a complex abstraction, it is often necessary to properly define a business domain. Reproduction is sometimes necessary for proper abstraction. Duplication reveals many use cases and makes boundaries clearer
• bad estimation. Quality takes time, not just skills. Intelligent programmers often overestimate their capabilities, and when time pressure begins to weigh on them, they use various hacks to complete the task at all costs.

“Premature optimization is the root of all evil.” – Donald Knuth

In general, programming is about favoring simplicity and practicality.

A good project is one that leads to simplified implementation and maintenance and makes it easier to understand the code. A design with a high level of abstraction and engineering is very likely to lead to difficulties in implementing new functionalities. It makes maintenance a nightmare and turns seemingly simple code into a winding maze of complexity. In addition, there is a direct correlation between the complexity and the number of potential errors.

How to avoid over-engineering?

The answer is very simple:

“Don’t write unnecessary code and look for simple solutions”.

So, to reduce the risk of excessive engineering, build a project with its intended purpose in minds. Carefully consider how it works, and what are the relationships between its individual components. Always keep in mind what you want to achieve at every stage of development and check your choices in terms of potential gains and losses.

Yes, it’s good to know the best practices along with their pros and cons, but not necessarily force their use. Instead, decide in each case whether to apply them or not, keeping in mind the benefit of both the company and the client. There is no single rule for all situations, each project is different, each client has different requirements, different budget, different deadline, the client and the contractor are guided by different values. That is why there is no one way to meet the needs of every business, not in such a dynamically changing sector as IT.

“Any fool can write code that a computer can understand. Good programmers write code that humans can understand. “
– Martin Fowler

So what techniques should be used to avoid over-engineering?

1. Simple solutions
2. Compliance with standards
3. Predictability of solutions
4. We’re all inclined to over-engineering, so be aware of that
5. As a rule, a more experienced developer writes cleaner, simpler code than a less experienced developer
6. It is worth consulting a solution idea with other programmers
7. Getting the job done is the most important, overarching goal. If best practices stand in its way, we should not follow them. This may sound a bit radical, but we should understand that the whole idea of best practice is to find the best possible way to develop software. Therefore, it makes no sense to use best practices that do not help us achieve this goal
8. We must understand that there is no silver ball. This means that there is no single method that can be applied in all situations. Most of us understand this, but for some reason we forget about it when we hear the words “best practice”
9. Do not design too far into the future if there are no clear requirements or premises for doing so
10. Our goals should determine how we specialize in our project
11. First write your tests and only in the second step the code – TDD technique
12. The use of refactoring in the event of over-engineering allows you to overcome redundant and redesigned solutions
13. “Done is better than perfect” – ugly working code is better than perfect but not working.

How do we, as software developers, measure the quality of the software we write? In most situations, the best measure of software is the value it provides.

The meaning of value depends on who our customer is. Once we understand who the customer is and what he wants to achieve, we will be able to assess what is value. Listening to the client will help you define what you want to achieve. The next step is discipline – with each new line of code you want to write, ask yourself – how much value does this line of code add to your customer’s product?

If the answer is “zero” – do not write this line. If it doesn’t add value, it probably reduces it. Whether by increasing the complexity of implementation, wasting customer money or delaying the delivery of other, more valuable functionality.

Over-engineering vs business

First of all, each new feature implemented for the client should work. The fun of over-engineering comes second – if time allows, that is. If we create a short-term product, it makes no sense to use practices that are used or even needed in large and very large projects. There is a huge likelihood that we will lead to over-engineering and disastrous effects.

Let’s not spend time and money on something that the client doesn’t need at the moment. If the client is waiting for an MVP, let’s focus on the MVP, without redundant implementation. In this way you will avoid implementing features that will never be used: Continuous Integration, static code analysis, several types of tests – all this is important but does not give the customer any value at any given time. These tools are obviously needed, but they must come at the right moment in the project. It happens that first a code is created that provides the customer with value and only then we put CI, write tests, introduce layers into architecture, CQRS, etc.

Let’s try to refer to the metaphor of pain – as programmers, we are to help our client deliver the product as soon as possible and at the cheapest price. Let’s design the application architecture in such a way that it organizes certain things sufficiently at a given moment. Architecture is to help us and not burden us with excessive complexity. Let’s choose solutions that are known to us and certain, we use ready-made solutions if they exist.

Using best practices is a great way, but these great practices can become dangerous when misused or misunderstood, which is unfortunately quite common.

Most importantly, the culture of overengineering blocks progress and productivity and is the reason why especially larger organizations have problems with performing tasks. While most of them claim to be “agile”, the practice of excessive engineering counteracts all agility and is its natural enemy.

Summary

Overengineering is not always something bad, it is often a conscious action, e.g. in the field of security. The reason for conscious over-engineering is also the care for quality and durability. Some projects actually include over-engineering in the development process, because such is the business need.

From the perspective of education and preparation for the profession of engineer – you can seek knowledge from technical books perhaps go to university and learn dozens or hundreds of engineering techniques and best practices. On the other hand, only experience can teach moderation and help you stick to actual problem solving.

The best place to start the fight against over-engineering is to follow the team’s core values. Do we value providing the customer with functionalities as soon as possible? Or maybe the most important thing for us is delivering error-free code, regardless of when it is delivered? It is worth to jointly define where the company is heading, and discuss the possibilities for scaling the team in a way that allows taking all this into account when proposing new solutions. On the one hand, it could provide the necessary resources to fight such problems as over-engineering. On the other hand, based on the company’s values, it would help being perceived by the clients as a mature and trustworthy organization.