Architecture Rituals and Patterns: On the Importance of the Middle Way Once and Only Once Balances Yagni - Part 2

...he heard a master teaching a girl how to play a string instrument. The master told her that if the string was too loose, it would not sound, but if the string was too tight, it would break: the string had to be at just the right tension to produce music and harmony. At that moment, he understood the middle way...

In the previous part of this series, I told you about YAGNI and OAOO and said that OAOO and YAGNI are like Yin and Yang. They balance each other. Without OAOO, YAGNI would slowly strangle us. With OAOO, you always have a system that you can improve (a phrase by Ron Jeffries).

I also "tied the cat," built systems as I saw others build them, was at the Shu level of ShuHaRi when programming enterprise systems, put my DAOs, put my Service, and put my Controller.

All the "well-architected" systems I knew did this, so I did it too. If a well-architected system looks like this in the end, it must be because it looked like this from the beginning... right?

Well, no. Things don't look the same when you're building them as when they're finished. If, for example, you try to build an arch without extra supports (which you will later remove), you could wrongly conclude that if you can't build it "without using supports," you're doing it wrong... That's not the case.

I discovered I was wrong almost by chance, I had already read that OAOO and YAGNI are like Yin and Yang. They balance, but I hadn't understood it beyond the sound of the words in the phrase... When I understood it, I wrote that article about epicycles.

I learned it when I came into contact with PHP programmers...

Speak your truth quietly and clearly; listen to others, even the dull and the ignorant; they too have their story. Desiderata

Those PHP programmers didn't know what YAGNI was... they knew nothing about the principle of building the simplest thing that could work (KISS)... they had never visited c2, and some of them had difficulty reading English...

And yet...

Yet precisely because of that, they followed YAGNI and KISS to a fascinating extreme, they did absolutely nothing that wasn't essential to finish on time the work assigned to them. Yes, their code was almost unbearable to the sight of someone like me at that time...

But what I needed months to build, they did in weeks; what I needed weeks for, they did in days; and what I needed days for, they did in hours...

On the other hand, their intense (and intuitive?) adherence to YAGNI and KISS also had its price, the technical debt they accumulated was brutal, and while they could put functionality in the hands of the user in record time, they also had to redo everything almost from scratch if they were asked for a change, there simply was no reuse...

There was no balance... But for me, it was a shock to look at the other extreme, and the other extreme looked back at me... I could spend hours refactoring code to eliminate redundancy to the maximum, whether the system was ready for the delivery date was a secondary factor... I was all OAOO... they were all YAGNI... I like to think that we learned a lot from each other..

Yagni means that we should only implement what we need now, not what we foresee that we are going to need. This strategy (when it works) eliminates some costs because it does not allow them to come into existence (they were things that we actually never came to need), and it delays the costs of those things that we will eventually need to a point in a future date. However, a commonly used counter-argument is that it is better to implement generic things now than in the future...

If we implement everything incrementally by invoking YAGNI, the system can become increasingly difficult to modify, especially if every time we implement something new, we do it always in the simplest possible way

It is good to do the simplest thing that could work, however, we must remember to reapply that rule to keep the whole system as simple as possible. This means that we will refactor the code so that nothing violates OAOO.

This refactoring, by centralizing any functionality into a single class or method, leaves the system well positioned for the next change. Anything you need to do afterward, you can be almost sure that you can do it better by improving or replacing a specific and isolated point of your code.

Now let's see an example:

• I have to make a registration screen, which includes
  

  a <select> element that displays the countries of the world, so that when a person registers on my site, I can present them with content relevant to their country (or countries) of origin. As I'm going to apply YAGNI, I won't do more than this:

• Controller:

public List<Country> getCountriesList() {
    logger.info("Getting countries!");
    List<Country> countries = getSimpleJdbcTemplate().query(
        "select id, name from country", 
        new CountryMapper());
    return countries;
}

Following YAGNI, I don't have to do more. If this ends up being the only screen in my entire system that displays countries, I won't have wasted my time on complexity without benefits.

But then, it turns out that in another part of the system, on the screen to register orders, when I have to put in the address where I want the product I'm purchasing to be delivered, and among the information to indicate in the address, I have to include the country

It is then, and only then, -when I already have 2 conceptually separate places, and for which it is not convenient to use the same controller- that I add a service:

• Countries Service:

public List<Country> getListOfCountries() {
    logger.info("Getting countries!");
    List<Country> countries = getSimpleJdbcTemplate().query(
        "select id, name from country", 
        new CountryMapper());
    return countries;
}

• Controller: User Registration

public List<Country> getProductsToList() {
    return getCountryServices().getListOfCountries();
}

• Controller: Order Address Registration

public List<Country> getProductsToList() {
    return getCountryServices().getListOfCountries();
}

This is why when a system is finished, it gives the impression that "everything is built with layers from the beginning." But that's a mistake, a product of seeing the system as something static, as something that "was born" with the form it had at a moment in time, and not as something organic, whose structure changes as it adapts to the needs of the user who acquired it.

The architecture of the project is not something that is predefined before understanding the system. The architecture is an emergent structure, derived from the user stories that are gradually built as the system is shaped. If the system does not include among its stories the need to operate on multiple different databases (Oracle, SQL Server, etc.), then the DAO layer should never have to exist. If it doesn't add value, it's superfluous.

In the case of the previous example, we can go even further. Nowadays, most applications are RIAs built on HTML5 and JavaScript, where the logic in Java (or in C#, or in Groovy, or in Scala...) merely acts as a layer of data exchange and security between the persistence layer (typically a database) and the business logic and presentation layer (which are usually located entirely in JavaScript)

So why do we insist on using POJOs that don't add value? Why doesn't our code look like this:

public List<Map<String, ?>> getCountriesList() {
    logger.info("Getting countries!");
    List<Map<String, ?>> countries = getSimpleJdbcTemplate().queryForList("select id, name from country");
    return countries;
}

Why complicate our existence with one (or several) POJO (or DTO) if our system is so simple (at the Java level) that the methods exposed by our controllers only do CRUD and it is the JavaScript code that really brings the data to life? What's the point of working with POJOs, if our domain model is anemic?

Approaches like OData, sweep away in one fell swoop all this paraphernalia of making controllers to slowly and step by step build a query API that could be automatically derived from the data model. But OData is just a query mechanism, which comes to eliminate the need to keep "tying the cat"... that is, to make a custom layer of controllers each time.

Why don't Oracle, SQLServer, MySQL, PostgreSQL, etc. directly offer an OData API (or something similar) instead of forcing us to reinvent it each time?

It seems that our human traditions make us live in cycles within cycles:

1. We invented Cobol, to expose our data through procedures.
2. We invented relational databases (whose implementation we never completed) to allow access to data through query expressions (SQL).
3. We then invented stored procedures, to procedurally encapsulate our query expressions.
4. We then invented object-relational mappers, to achieve compatibility between multiple databases and be able to query them with a unified query language.
5. We then invented SOAP, to procedurally encapsulate our objects.
6. We discovered that in parallel, someone had invented REST, which allows us to query resources using query expressions represented by a URL.

And so... again and again, procedures, expressions, procedures, expressions, procedural approach, declarative approach... and every time we reinvent the wheel, we feel like we are breaking new ground... when in reality, we are just giving the wheel of fashion another turn.

Will we ever find the balance? (Or perhaps we have already found it... and just need to realize it...)

 Originally published in spanish in Javamexico

 

 

Architecture Rituals and Patterns: On the Importance of tying up the cat - Part 1

The Zen master and his disciples began their evening meditation. The cat that lived in the monastery made so much noise that it distracted the monks from their practice, so the master ordered that the cat be tied up during the entire evening practice. When the master died years later, the cat continued to be tied up during the meditation session. And when, eventually, the cat died, another cat was brought to the monastery and was tied up during the practice sessions. Centuries later, scholars descendants of the Zen master wrote treatises on the profound spiritual meaning of tying up a cat for meditation practice.

A few years ago, I attended an event organized by Sun to promote Java here in Mexico (I think it was called Java Dev Days). At that time, I entered a presentation, I think it was by Sang Shin (the creator of JavaPassion), in which he explained how simple it was to use JRuby with Ruby on Rails to create a "hello world" page. It was something like this video: http://www.youtube.com/watch?v=sas3KCKwyHI. It was the first time I saw RoR, so on that side, it seemed educational... however, most of the audience... was falling asleep!

That caught my attention... why were they all so bored, from what I had been able to observe the previous days, many of them were enterprise programmers, with the kind of clients that I have had, who give you very little information to work with, demand a lot, and want everything for yesterday..., so I started to listen to the comments that they exchanged quietly (or not so quietly) between yawns:

    Is that simple? It took like 15 minutes to make a hello world screen!
    How boring, I could do that with JSP in 1 minute
    Where does this guy get that this is simple? Did you see the mega directory structure that those commands created? That's not simple at all!
    You can tell this presenter spends his time giving courses, but he's never worked in real life, I'd like to see him with his RoR having to get the job done at 3 in the morning

At least the 2 rows sitting in front of me were not remotely impressed with RoR. And why would they be if in JSP you can make a hello world simply by putting something like this:

<HTML>
 <HEAD>
  <TITLE>Hello World</TITLE>
 </HEAD>
 <BODY>
  <H1><%= "Hello World!"  %> </H1></BODY>
</HTML>

Certainly, arguing that the 15 minutes of the multi-step video are simpler than this is absurd.

Well, some will say, (as part of me thought) the point of the example with RoR is not to show how simple it is to make a silly "Hello World" but to show the potential of RoR's MVC architecture to do much more complex things...

Let me see if I understood... are you telling me that I'm going to complicate my life by making something easy much more difficult and that's going to result in a simplification of the difficult? More complexity today == Less complexity tomorrow? That violates YAGNI. (Yagni means: You aren't going to need it)

And it's not that you have to avoid violating YAGNI just to follow tradition, as in the case of the cat in the monastery. Yagni has a reason for being: in systems development, change is the only constant, and if you get ahead and build something before you have a clear idea of whether you really need it, when you get to the point where you are really going to use it, it is very likely that what you really need will be very different from what you had predicted. This doesn't mean that your code shouldn't be flexible, but you have to be very careful, because you can end up with an over-engineered solution, ready for situations that will never occur. If you use too much energy today, fighting with the ghosts of things that will never come, when the real enemies finally arrive, you won't have the strength to face them.

Now, one of the big problems we have in learning programming is that when we are learning, we have to build examples like a Hello World in RoR to understand how it works. But as instructors, we have to be very careful in saying "look, with this example I'm going to show you how simple this is...". No. You don't make a hello world in RoR because it's simple. Let's not confuse our students. We don't go to the gym to sweat because it's less tiring than staying at home watching TV. We go because by doing so we will improve our skills, we go to prepare ourselves for future situations. So, in programming, not only Yagni rules, we also have another principle: OAOO. (OAOO means, once and only once).

OAOO is the root cause of the apparent complexity of RoR, and the archenemy of copy&paste. OAOO has other names, it is also known as the DRY principle (Do not repeat yourself). It's not good to have copies of the same code scattered throughout the project, because if there is a bug in one of them, and you didn't follow OAOO, you'll have to find and fix them all, instead of solving the problem in one place.

OAOO and YAGNI are like Yin and Yang. They balance each other. Without OAOO, YAGNI would slowly strangle us. With OAOO, you always have a system that you can improve (a phrase by Ron Jeffries).

OAOO alone would also kill us, leading us to an eternal refactoring of code searching for the holy grail of the generic code with the fewest possible lines without ever stopping...

We need both. We need balance.

How many times have you faced code like this in your work:

DAO:
    public List<Product> getProductList() {
        logger.info("Getting products!");
        List<Product> products = getSimpleJdbcTemplate().query(
                "select id, description, price from products",
                new ProductMapper());
        return products;
    }

Service:
    public List<Product> getListOfProducts() {
        logger.info("Getting products!");
        List<Product> products = getProductsDao().getProductList()
        return products;
    }


Controller:
    public List<Product> getProductsToList() {
        logger.info("Getting products!");
        List<Product> products = getProductsServices().getListOfProducts()
        return products;
    }

What value does this add? Why can't we simply do it like this (no Service layer, no DAOs layer):

Controller:
    public List<Product> getProductsToList() {
        logger.info("Getting products!");
        List<Product> products = getSimpleJdbcTemplate().query(
                "select id, description, price from products",
                new ProductMapper());
        return products;
    }

Why is it that if you don't tie up the cat before meditating... I mean, why is it that the principles of multi-layer architecture dictate that it should be done this way... really? Is that really the answer? NO!

If it doesn't satisfy an immediate need and doesn't help OAOO (obviously the redundancy in this doesn't help OAOO), we shouldn't do it. We are burning precious time and energy on "busy work" that nobody will thank us for.

Heresy! Many have told me when I tell them to stop having so many layers just because. Layers are NECESSARY. You can't remove them.

Of course, you can, and I'm not the only one who thinks so. JBoss Seam was designed precisely with the idea of eliminating so much unnecessary layering. And that was YEARS ago. Why hasn't it dawned on most people that making layers for the sake of making layers is not correct? Why do we still tie up the cat and buy a new cat every time we start a new project? Tradition.

Now, traditions are not all bad, (those cultures with traditions that don't help their survival simply become extinct). Where does the idea come from that putting so many layers helps in any way?

I will talk about that in my next post in this series...

(originally published in spanish on Javamexico)

Estimation: Performance Assumptions

A very common request (which often becomes a demand) made by clients is a performance guarantee, things like:

• I want all screens/pages to have a maximum response time of 3 seconds.
• I want all queries to have a maximum response time of 3 seconds.
• I want the system to run on my server (when we're lucky, they tell us the server's specifications).

What to do in such a case? When I had just graduated (more than a decade ago) my response used to be: "I need more data", "With the information I have, I can't guarantee that performance", "I don't even know what the algorithm for the indirect sales commission calculation process consists of, how am I going to guarantee that the screen where I see the result responds in 3 seconds???"

It almost goes without saying that my attitude was the wrong attitude...

My questions seemed logical to me, to give a guarantee of a page's response time of 3 seconds was like being told "you're going to transport a box of indeterminate weight, across an unknown distance, using an unknown mechanism and you have to guarantee that you can do it in 3 seconds".

Of course, my response was: If I don't know the weight, the distance, or the mechanism, I can't guarantee the time....

And the inevitable response from my bosses: Don't tell me why not, tell me "what would have to be true to make it happen"... was irritating to my ears.

But they were right, if you're asked for a guaranteed transport time, and you don't know the weight, the distance, or the mechanism to use, what is expected is not that you say "I lack that data", what is expected is that you say under what conditions you could do it in the specified time (if later the client cannot provide them, then it's not your failure).

Of course, this doesn't mean you shouldn't ask. It's always best to ask. But paradoxically, you should ask, but not worry too much whether or not you get an answer. There will be clients who are interested in you building the right thing. Those clients will try to answer your question by giving you the parameters you require. There will be other clients for whom the "right" thing doesn't matter, the only thing that matters to them is that it's done "correctly" (that it's well done, even if it's not useful to them). This second type of clients will often be irritated by your questions, they might even answer "you're the expert, I don't know about this, you tell me what you need to do it in the time I require"... There will be cases where your client is lucky, and the right thing and the thing done correctly will coincide... but always try to do the right thing, but if your recommendations are ignored, don't get irritated, and switch modes. Don't think that your client is "stupid" or "bad" for demanding the "what needs to be true". Ultimately, they are right, you are supposed to be the expert: tell them "what needs to be true", and let them decide whether or not they have the budget.

Thus, all the missing parameters must be assumed. Let's take the most extreme case, they ask for a response time of 3 seconds, and they give you no other clues... and there are various requirements in the system (such as "display commissions for indirect sales"). You don't know how many sales they have, but you know it's a multinational corporation, and they are certainly not "a few."

What to do? Let's go back to the example of transporting a package:

Weight: Unknown

Distance: Unknown

Mechanism: Unknown

Transport time: 1 hour.

What would you do?

Well, you could, for example, say: In the basket of my bicycle, fits 1 liter of water. And in one hour, on my bicycle, I cover let's say 40 kilometers, then:

Weight: 1 kilo

Dimensions: maximum 15 cm square (the volume that fits in the basket of the bicycle)

Distance: maximum 40 km

Mechanism: Bicycle

Transport time: 1 hour.

Now, the big mistake why assumptions like these are considered the mother of all evils is because the one who assumes them keeps them until the day the client wants to send a package, and it turns out that half a ton of bricks needs to be sent 100km away in a maximum of 2 hours... and there's no way in the universe to do that in the basket of your bike...

And then, comes the typical sermon: "don't assume!" "ask!" (Which becomes particularly annoying because you clearly remember asking the client and they never gave you a clear answer)

Was the error in the assumption? NO.

The error was that you did not make your assumption PUBLIC. You didn't verbally discuss it with the client, you sent it by email and tried to get them to sign a document accepting it. Now, of course, some might think: "I would have to list all the possibilities" — what if the package is 100kg, what if it's 10 square meters, what if I have to deliver it in 15 minutes? That's the wrong approach. The correct approach is to give one example, but very clearly defined.

Of course, some will say, there will be something you overlook: what if the user asks to send 1 kilo of uranium? It meets all the criteria, but the radioactivity will kill you! There's no way to think of everything.

Indeed, there isn't. But that's no excuse not to try to define things as precisely as possible. If you strive to define your assumptions as best you can, they will gradually become more complete, and after a few years, experience will allow you to include enough factors so that the possibility of being asked to transport something that violates your assumptions and causes you harm is very remote. If at the beginning of your career as a developer, 9 out of 10 assumptions lead you into problematic situations, and 10 years later, only 5 out of 10 do, that is a success.

Returning to the scenario that interests us as developers, if you are asked: “I want all queries to have a maximum response time of 3 seconds”

Establish assumptions:

• Maximum number of records per query.
• Maximum data weight of the query in Megabytes.
• Network speed (Mbits/s).
• Hardware characteristics of all servers and client workstations involved.
• Type of operation (if it's in SQL, maximum number of records, presence of indexes, maximum number of records in the tables, maximum number of joins, version of the pseudo-RDBMS, etc., etc., etc.).

In civil engineering, if an engineer wants to know the strength of a steel bar 10 cm thick and 10 meters long, they can look it up in a catalog...

Isn’t it time we built that kind of knowledge for ourselves?

As much as I've looked for tables with the above data, where someone has tested certain "typical" hardware/software configurations and basic algorithms”, something like:

If you perform a Join between 2 tables with such and such columns in Sql Server 2008R2, on an i5 server, with 8GB of RAM and a disk of certain characteristics, on a network of... and its combinations (after all, we are programmers, we create algorithms, let's make one that generates X combinations).

Will it be a benchmark that works absolutely? Of course not, but that's not the goal, the goal is to be able to offer a reasonable assumption, and give your client the confidence that if the conditions you set are met, you can guarantee a result.

Then, if something else happens along the way... well, it's not your fault, you couldn't control it, and the number of clients who will understand that you didn't meet the goal because they (or the circumstances) violated your assumptions is (at least in my experience) much greater than those who would understand if you tell them from the beginning "why not" instead of "as if."

Remember, assumptions are not bad. If you ask a civil engineer to build your house, you don't expect to answer their questions about the strength of materials, that's not your job. The job of the consultant (whether a software developer or a civil engineer) is to find the conditions under which "it can be done," and in software, fortunately... or unfortunately, almost everything can be done, if you have the right assumptions.

 Originally published in Javamexico

Estimation: Assuming is good, we must assume as much as possible

 

Before continuing with more concrete examples of assumptions, I find it important to tell you that when I started in software development, I often encountered this situation:

Project Leader: The client said that feature XXX doesn't seem useful to them. They don't understand why we made it this way when they needed something else. 

Programmer: Well, I assumed that... 

Project Leader: Well, next time don't assume: ask! Programmer: Sorry, it won't happen again.

And a few days later:

Programmer: How should we make feature YYY? 

Client: Well, in such a way that we maximize efficiency and add value to the business... 

Programmer: Yes, but, is the relationship between products and orders one-to-many or many-to-many? Client: (What is this guy talking about?) Sorry, I have another meeting, we'll look at it later. 

Programmer: But but...

And finally:

Project Leader: The client said that feature YYY doesn't seem useful to them. They don't understand why we let them down again, they needed something else. 

Programmer: I did ask, really, but he only explained it very superficially. 

Project Leader: Well, next time, get detailed information. Programmer: Sorry, it won't happen again.

And so, always caught in the same loop, without control, sometimes doing well, sometimes not, depending on how much the programmer's intuition matched the client's reaction to the tested software.

Sadly, what the programmer learns after several such cycles is that assuming is bad. Wrong conclusion. Assuming is the best thing we can do at the start of a software project. Let's assume everything we can possibly assume. But let's be explicit about our assumptions.

Let's change the dialogue, negotiate differently, so it looks like this:

Programmer: How should we make feature ZZZ? 

Client: Well, in such a way that we maximize efficiency and add value to the business... 

Programmer: Here I have (many detailed and explicit assumptions) a prototype of the screen, here is my user story and I've already written some acceptance criteria, please give them your approval so I can start programming. 

Client: Ok, I'll review them later and let you know. 

Programmer: Perfect!

Of course, one of the most common problems we might encounter is that the client finally doesn't have time to review and never gives us approval or does give it, but without really reviewing the assumptions.

Project Leader: The client said that feature ZZZ doesn't seem useful to them. They don't understand why we made it this way if they needed something else. 

Programmer: Well, I delivered a prototype, the user story, and the acceptance criteria, and he gave approval, if you want we can review them to see if they correspond, but the Tester already has evidence that there is correspondence and all the bugs have been fixed. 

Project Leader: ... Well, we'll have to tell the client that if they want it different, they need to request a change.

This is the point we want to reach. Yes, the best we can do, the ideal that the agile movement aspires to, is to build what the client really needs, but often it is not achieved on the first attempt, because the first who doesn't clearly know what they need: is the client. And the technique to get there is assumptions. But explicit, visible, clear assumptions, if they are about screens, with prototypes, if they are batch processes, with examples of congruent input and output data.

The bad thing is not the assumptions, the bad thing is when we assume "in secret", without evidence, without giving visibility.

In the next part of this series, we will continue exploring more vaguely defined assumptions and how we can turn them into refutable, bounded assumptions, which, when violated (by the client), result in more work and paid time (remember, the goal is not to finish the project, software is like a plant, like a living being, the goal is to cultivate it, let it grow, mature, adding value to the client if possible, indefinitely).

We must strive to do the right thing, without forgetting to do it correctly.

Originally published in Javamexico

Estimation: Negotiation and the difference between doing the right thing vs. doing it right

In English, there's a saying: "There is a difference between doing the right thing and doing the thing right."

In software estimation, and the negotiation process necessary for a consultancy to build software for a client, the difference between one and the other is tremendously important.

To "do the right thing," we must clearly understand what the client really needs (regardless of what they ask for), whereas to "do it right," it's not relevant whether what we do will actually be useful to the client or not; what's important is whether what we did is well-built.

An extreme example:

A client comes to you and describes a single-passenger transport vehicle that allows them to travel anywhere. You build them a bicycle, but they were thinking of using the vehicle in Alaska, in the snow—they were looking for a snowmobile.

Ultimately, it doesn't matter how well you built the bicycle (done correctly) because it's not useful to the client since what they needed (doing the right thing) was for you to build them a snowmobile.

During the negotiation process carried out during the construction of software, the difference between "doing the right thing" and "doing it correctly" has an important peculiarity: Who has the information required to achieve one objective or the other changes drastically.

For "doing it correctly," all the knowledge is with the consultancy that builds the software, in the technicians, the developers, the designers, who, if they are good, will make the most usable screens, and will use the best architecture and algorithms, strictly adhering to a process that helps ensure a well-built product.

Conversely, for "doing the right thing," the information is with the client. Only they can, although often not at the beginning of the software development, determine if what is being built for them is ultimately what they need. Yes, and only if the client is willing to go through this process of self-discovery, in which their assumptions will be carefully examined, can they finally conclude whether that nebulous idea in their head is really what they need, or if instead, they should have asked for something else from the beginning.

Of course, the consultancy, with its analysts, developers, architects, testers, etc., must seek to facilitate dialogue with the client and help them quickly enter the cycle of assuming, implementing, validating, and re-assuming until they construct "the right thing," but it's important to remember that ultimately the information comes from the client, not the consultancy, and if the client, due to some problem of their organizational culture, is not willing to invest the time required to clearly define "the right thing," the consultancy will not be able to build it (you can lead a horse to water, but you can't make it drink).

There is a prayer that says "God, grant me the serenity to accept the things I cannot change, the courage to change the things I can change, and the wisdom to know the difference." Well, in this case, we need to be very clear about the difference: Only the client has the power to help us build "the right thing," the consultancy is just a facilitator, it's very important to remember this because when we are constructing the proposal, we must ensure it helps to build "the right thing," but at the same time, our responsibility must not go beyond "doing it correctly."

I have had to deal with multinationals, who give me just a few hours to prepare a proposal for months of development. The amount of effort wasted in such developments is absurd, and yet, it was useless for me to insist that 3 to 6 hours are not enough to estimate a project of 3 or 6 months duration, too much remains unspecified. At first, when I faced this situation, my attitude was "it can't be done," "avoid assuming," "ask the client for the missing information, or if not, do nothing." Sounds very nice from an abstract perspective, but that attitude doesn't pay the bills... so I changed it.

I still tell the client whenever I can that the time they are giving me is not sufficient, and that it is very risky to do a project with so little time to plan, but once the client indicates that my warnings are falling on deaf ears, I immediately change strategy: This client isn't interested in me "building the right thing," what they care about is having something built that "comes as close as possible to that," and I'd better "do it correctly," because in the end, if the approximation isn't close enough to what the client realizes they needed to receive (minutes after receiving it), recriminations and attempts at penalties will follow.

Fortunately, "doing the right thing" is subjective, whereas "doing it correctly" is completely objective. If you needed a snowmobile, and I built you a bicycle, and the very limited assumptions of the proposal are met by the bicycle, there will be no way for you to penalize me, or refuse to pay me, and in the event of a lawsuit, you will lose, because I will be fulfilling the contract.

The Agile Manifesto says "Customer collaboration over contract negotiation," which is another way of saying "prefer doing the right thing over doing it correctly," but the Agile Manifesto clarifies: "while there is value in the items on the right, we value the items on the left more." It's the same with "doing the right thing" vs. "doing it correctly," both are valuable, and certainly the best would be to do both, "the right thing, correctly," but we must have the wisdom to know the difference between what we can change... what is in our hands, and what is a shared effort, and that if the other party outright refuses to cooperate, it will not be possible to achieve.

It is very important to do this with a cool head and an enthusiastic and cheerful attitude. If a client is difficult in the first place, and we also treat them rudely, we will get nowhere; the warmth of kindness has more impact than the coldness of aggressiveness. As much as possible, we should seek common ground, empathize with the client, and move forward together, but we must also have clear boundaries drawn so that if things get out of control, we can cut cleanly, without irreparable losses to the continuity of our business, otherwise, we'd fall into Stockholm syndrome...

Don't miss the upcoming episodes of this exciting series on estimation, where we'll see different examples of real-life situations you might face, and how you can emerge victorious from them...

Update: I came across this article on InfoQ, which I see as very related to the topic I'm discussing here and I loved this example, so I take the opportunity to share it with you:

People buy a hammer to drive a nail to be able to hang a picture - they know they can achieve their goal (hang the picture) with the acquisition of the hammer.

Unfortunately, in the context of software development, things are not so straightforward between what is delivered (the software) and the business objective that is desired to be achieved. Many people do not even try to reach the business objective. This creates a significant risk that the provider only delivers what the client asked for - software that satisfies a vague set of requirements - instead of what the client really needs, which is to achieve their business objective.

Unfortunately, it is the same clients who tell the provider: Do exactly what I asked, stick to the contract, do it correctly, I have already decided that this was the right thing... without realizing the harm they are doing to themselves. As providers, we have the responsibility to ask: Why do you want me to make you a hammer? But if the client ultimately refuses to explain the purpose, we must accept that the client is ultimately responsible for their own well-being, and make their hammer, even if their plan is to hit themselves with it... imagine if knife manufacturers spent their time stressing about "what if someone cuts a finger," or stove manufacturers about "what if someone gets burned," practically all industries would end, since there is no limit to the misuse that can be given to things...

Clients are not children, and while we must clearly warn them of the consequences of their actions, the responsibility for their actions lies with them.

In the next part of this series, I will talk about techniques for dealing with performance requests (or demands) in situations where apparently there is not enough contextual information.

Originally published in Javamexico

Estimation: Is assuming the mother of all estimation errors? Or perhaps it is "to suppose"? Or maybe it is just our own ignorance?

I recently read a tweet that said:

Congratulations to "Assuming" for being the MOTHER OF ALL ESTIMATION ERRORS in software development!!!!

— SoftwareEvangelist (@vanessa_amaya) May 10, 2013

And I wondered: does the author realize... that she is assuming that assuming is the problem?

First, let's review the meaning of the word:

assume: assume. (From Lat. assumere).

1. tr. To take to oneself, to take for oneself.
2. tr. To take charge of something, to be responsible for it, to accept it.
3. tr. To acquire, to take on a greater form.

In software, the meaning we commonly use based on my experience is "To take charge, to be responsible for something, to accept it." What do we take responsibility for? The assumptions we use to build our estimate. Should we instead seek to be irresponsible? In my opinion, the problem is rather the assumptions themselves. If the assumptions do not align with reality, we end up delivering things poorly, for example in the scenario Ezamudio discussed in the previous post of this series, the problem escalated because the technician assumed he had chosen the correct part; in software development, it is more complicated, as there are several intertwined assumptions. But returning to the beginning, is assuming really the mother of all problems? No. Is supposing then the mother of all problems? I don't agree with that either, let's review the meaning of suppose:

suppose. (From Lat. supponere).

1. tr. To take for granted and existing something.
2. tr. To pretend, to give an ideal existence to what really does not have it.
3. tr. To entail, import. The new acquisition he has made implies excessive conservation expenses.
4. tr. To conjecture, to calculate something through the indications one has.
5. intr. To have representation or authority in a republic or in a community.

The meaning we commonly use in software, (at least those of us involved in its construction) is to conjecture, to calculate something through the indications one has, while we consider that the client sees it more as "To take for granted and existing something." Contradictory? Superficially, one might think so, but if examined more deeply, there is no contradiction. Supposing is a necessary something, we get out of bed, and we suppose we are awake, we drink water from the jar and suppose it is not toxic, we go out and drive to work and suppose it will be relatively safe. We do not have absolute certainty, but we also do not torment ourselves, we act taking it for granted, and if in the end the water or food causes us indigestion, we deal with it, if we start having indigestion several days in a row (if it does not kill us) we will change our diet and the water we drink. The same applies to software assumptions.

Supposing is a necessary activity in the scientific method (step 3):

1. Observation: Observing is applying the senses attentively to an object or a phenomenon, to study them as they really present themselves, which may be occasional or causal.
2. Induction: The action and effect of extracting, from certain observations or particular experiences, the particular principle of each one.
3. Hypothesis: Formulation through observation following the norms established by the scientific method. <---- ASSUMPTION!
4. Test the hypothesis by experimentation.
5. Demonstration or refutation (antithesis) of the hypothesis.
6. Thesis or scientific theory (conclusions).

Basically, we Observe, Analyze, Assume, and Test. If it works, we keep the assumption, if not, we repeat until we find the "correct" assumption. Now, this is the most important thing about the scientific method: We never reach the correct, the path to truth is like the limit in infinitesimal calculus, we are getting closer each time, but we never arrive.

The most important lesson of the scientific method is implicit: It's a Sand Mandala. Assumptions are made, only to be replaced by better ones. Thus, in software, assuming our assumptions are true is not a mistake, it's part of the method; the mistake comes when we forget that we are just at step 3, step 4 is still to come, where, without neglecting our responsibility, we test if our assumptions match reality. Therefore, assuming and supposing are not the mother of all problems, the mother is rather the lack of experimentation, and this lack of experimentation stems from treating assumptions incorrectly.

The first rule to follow for assumptions, again a rule from the scientific method, is that the assumption must be refutable (principle of falsifiability), what does this mean? There must be a concrete way, through a particular observation, to declare the assumption (provisionally) true or false. (An rrefutable assumption would be one that requires an exhaustive search of all possibilities to refute it). For example, I've been given RFPs to make systems that say "the system must implement all the relevant business rules for the business process to be automated" and then, nothing, no list of which rules are. Demonstrating whether we are meeting or not meeting that assumption cannot be refuted, the list of rules is potentially infinite (and clients certainly take advantage of that). What can we do?

When I was younger, what I used to do was demand the list of rules, or say that I couldn't do anything and refuse to estimate the time that would be necessary. Needless to say, that attitude only got me into trouble with my bosses, lost business, and dissatisfied clients. Although that's the path indicated in estimation books, it's not the correct path in the real world.

Of course, there are clients who are willing to give you the list of rules, so when you're in this situation, don't fail to ask for the list, but if they can't (or won't) give it to you, don't suffer, accept it, and continue... but cautiously. A few months ago, I watched a nascent consultancy suffer a horrible death and enter zombie mode: they sold a system with open-ended assumptions, according to them, it was going to be finished in 2 weeks; when I met them, they had been working for free for the government for a year, under threat of a lawsuit for breach of contract if they were to "crack" and not finish implementing all the rules that the client might think of. I couldn't help them anymore, what they needed was a good lawyer. How then to avoid falling into that hole?

Some would say: Simply reject the business. Yes, it sounds nice when you have a lot of money, have no responsibilities, and if you don't receive income you're the only one who doesn't eat. But what if your family depends on you? Can you really afford to reject the work? Of course not. Don't reject it then, nor tire yourself explaining why you can't "implement all the relevant business rules for the business process to be automated", the right answer is: define the scope yourself.

Place an assumption in your proposal in the following way: "It is assumed that these business rules will not exceed 10. It is assumed that the list of all business rules to be implemented will be received X days after the project has started. The rules must be defined as algorithmic step-by-step procedures, each consisting of no more than 10 CFP (Cosmic Function Points according to the standard ISO/IEC 14143/1:2003). It is assumed that all the information to be used by the business rules is that which is fed into the system through the user stories defined in the scope section of this proposal. If the list of business rules is not received at that time, or if a greater number of rules are received, the scope change procedure described in this document will be executed, which may have an impact on the project times and costs." There, you've defined the scope.

By doing this, essentially you are answering the question that some clients would surely ask you when you told them that you couldn't "implement all the relevant business rules for the business process to be automated". Some clients would ask: Why not? And you would tell them: Because I don't know how many there are, nor how complex they are, and you're only giving me 3 months to do them. Well, the client would say: How many can you do in 3 months?

And there is where, when I was younger, I would clash most strongly with reality: I simply couldn't answer that question! I could only say "because not", when the right answer is to say "as if". This is not a silver bullet, there will be clients who are not willing to accept a defined scope. In those cases, if the client is not willing to pay for time and materials, then yes, the right answer is thanks, but no thanks, I can't do the project.

In the next part of this series, we will delve into the benefits of assuming, and why the myth has been generated that one should not assume.

 Originally published in Javamexico

Estimation: Why do we do it?

 

There are those who consider that estimating is impossible and a waste of time, a useless endeavor. There are those who believe it can be done, but only under certain conditions, and there are those who think that the secret lies in following a certain method... However, before discussing these points of view, I want to focus on a question often omitted in articles and books on estimation:

Why do we estimate? And I'm not talking about the theoretical reasons typically used in books on the subject, but rather, out there, in the real world. I don't want to generalize, so what I will say next is strictly based on my own experience.

We don't estimate to know how long the project will take; the client usually has already set a deadline that is unlikely to change.

Why do we estimate then? We estimate to see if we can do something within the time and budget that are already established and that sounds to the client like what they asked for (because if one thing is certain, it's that most clients don't really know what they want until they've seen a couple of progress demos).

The agile community tells us: don't estimate, do everything by time and materials, although in reality what it's saying is "don't estimate beyond the next sprint" and "lose any client who is only willing to buy projects at a fixed cost and time." It would be nice if the world were that easy, but the fact is that it's not: most of the clients I've encountered in the consulting world want a budget upfront... Those of us who would like everyone to understand Scrum sometimes find it absurd, a display of their ignorance some say, but let's put ourselves in their shoes: who, when taking their car to the shop, expects to be told: we're going to fix your car iteratively, give us your bank card with an open voucher and then check your balance to see how much it was at the end...

And yet we go to the doctor, and while each visit may have a fixed cost, we don't ask the doctor to give us the cost of the medicines before diagnosing us, nor do we force them to tell us the maximum number of consultations we'll need... Medical treatment is completely iterative... Maybe one day we'll recognize the similarities between curing the most sophisticated biological system we know and creating or composing new systems... In the meantime, let's think... Why are there fixed-cost projects that do work? The level of uncertainty clearly visible to the educated mind at the start of the project should guarantee that this never happens... What's different in those projects? Maybe it's not obvious, but in my experience, it's quite simple: the scope was better defined, either because the client themselves had already done some pre-analysis, or because the consulting team did the same... SACRILEGE! Am I daring to say that the correct way to carry out a project is with Waterfall?

NO! To believe that is to ignore the principle of diminishing returns: those who believe that this indicates Waterfall is the answer are making the mistake of thinking that if some analysis is better than none, then double or quadruple the analysis will bring a benefit directly proportional. Well, welcome to the real world: things don't work like that, if you do a lot of analysis, you also consume a lot of time and while your understanding of the situation becomes more precise, it also becomes progressively outdated... The world doesn't stop changing because you analyze a particular point of it in time and by the time you turn to compare your analysis with the current reality, the world has already changed: the conclusions of your analysis are now obsolete.

Zero analysis? Bad. Exhaustive analysis? Bad... What to do then? Find out how much is "enough" but first understand enough for what? Many books on estimation talk about achieving 75% accuracy, as the ultimate goal of a well-done estimate. In my experience, that perception is wrong. What good does it tell a client: to succeed in what you're asking, I need one million pesos, if the client only has (or claims to have) a third of that? It's no use, the client won't have their product, and we won't have our project. What to do then?

Look for the "as if," the first time I heard that phrase it seemed naive: do a one million project for a third? Impossible! But then it happened that some people asked me: why? And I complicated my life giving a thousand explanations about the complexities that are part of software development (many of which I didn't understand then... and many of which I still don't fully understand now) and... I couldn't convince them, and ended up trapped in horrendous Dead March projects from which I not only received pressure from the client, desperate because the project was nowhere near completion by the deadline, but also from the company I worked for, equally desperate as the project's margin eroded away...

And after finishing that project, we embarked on another just like it... How do we break that cycle? Is  it true, as I recently read on Twitter, that assuming is the mother of all problems in system development. I will talk about that in my next post...

 Originally published in Javamexico