Requirements Analysis: Negative Space

A while ago, I was part of a team working on a crucial project. We were confident, relying heavily on our detailed plans and clear-cut requirements. Every day, we would gather, discuss our progress, and tick off the tasks we had completed. We were so focused on what was written down, on the tangible and the explicit, that we neglected something vital—the negative space, the unspoken needs, and assumptions that lingered in the background.

As the project neared completion, we began to sense that something was amiss. Our system, though functioning as per the specifications, felt out of sync with the users' real-world needs. It was like a puzzle with all the pieces in place, yet the picture it formed was not quite right. We had overlooked the importance of listening to the message between words, of paying attention to what was not said.

In our meetings, there were hints of this negative space—subtle hesitations, unasked questions, and fleeting expressions of concern. But we were too preoccupied with our checklist to notice. It was only when we presented our nearly finished system to the stakeholders that the full impact of our oversight became apparent. The feedback was polite but pointed—our system, while technically proficient, missed the mark in addressing the unspoken needs that were crucial for its success.

This experience taught me a valuable lesson. In the rush to fulfill explicit requirements, it's easy to ignore the negative space. But it's in this space that the essence of a truly effective system lies. Since then, I've learned to listen more attentively, to observe not just what is said, but also what is left unsaid, and to embrace the wisdom found in the quiet spaces between words. In doing so, I've discovered that the most successful projects are those that harmonize both the explicit and the implicit, creating a system that resonates with the needs of its users in a way that goes beyond mere specifications.

The challenge of dealing with negative space is not only a human endeavor but also a significant hurdle for Large Language Models (LLMs) like GPT. While these models excel at processing explicit information, they struggle to grasp the subtleties of what's left unsaid, the nuanced hints and cues that humans can pick up on. Bridging this gap requires a blend of technological advancements and human intuition, a synergy that continues to evolve as we strive to create systems that truly understand and cater to the myriad needs of their users.

Architecture Rituals and Patterns: On the Importance of the Middle Way Once and Only Once Balances Yagni - Part 2

...he heard a master teaching a girl how to play a string instrument. The master told her that if the string was too loose, it would not sound, but if the string was too tight, it would break: the string had to be at just the right tension to produce music and harmony. At that moment, he understood the middle way...

In the previous part of this series, I told you about YAGNI and OAOO and said that OAOO and YAGNI are like Yin and Yang. They balance each other. Without OAOO, YAGNI would slowly strangle us. With OAOO, you always have a system that you can improve (a phrase by Ron Jeffries).

I also "tied the cat," built systems as I saw others build them, was at the Shu level of ShuHaRi when programming enterprise systems, put my DAOs, put my Service, and put my Controller.

All the "well-architected" systems I knew did this, so I did it too. If a well-architected system looks like this in the end, it must be because it looked like this from the beginning... right?

Well, no. Things don't look the same when you're building them as when they're finished. If, for example, you try to build an arch without extra supports (which you will later remove), you could wrongly conclude that if you can't build it "without using supports," you're doing it wrong... That's not the case.

I discovered I was wrong almost by chance, I had already read that OAOO and YAGNI are like Yin and Yang. They balance, but I hadn't understood it beyond the sound of the words in the phrase... When I understood it, I wrote that article about epicycles.

I learned it when I came into contact with PHP programmers...

Speak your truth quietly and clearly; listen to others, even the dull and the ignorant; they too have their story. Desiderata

Those PHP programmers didn't know what YAGNI was... they knew nothing about the principle of building the simplest thing that could work (KISS)... they had never visited c2, and some of them had difficulty reading English...

And yet...

Yet precisely because of that, they followed YAGNI and KISS to a fascinating extreme, they did absolutely nothing that wasn't essential to finish on time the work assigned to them. Yes, their code was almost unbearable to the sight of someone like me at that time...

But what I needed months to build, they did in weeks; what I needed weeks for, they did in days; and what I needed days for, they did in hours...

On the other hand, their intense (and intuitive?) adherence to YAGNI and KISS also had its price, the technical debt they accumulated was brutal, and while they could put functionality in the hands of the user in record time, they also had to redo everything almost from scratch if they were asked for a change, there simply was no reuse...

There was no balance... But for me, it was a shock to look at the other extreme, and the other extreme looked back at me... I could spend hours refactoring code to eliminate redundancy to the maximum, whether the system was ready for the delivery date was a secondary factor... I was all OAOO... they were all YAGNI... I like to think that we learned a lot from each other..

Yagni means that we should only implement what we need now, not what we foresee that we are going to need. This strategy (when it works) eliminates some costs because it does not allow them to come into existence (they were things that we actually never came to need), and it delays the costs of those things that we will eventually need to a point in a future date. However, a commonly used counter-argument is that it is better to implement generic things now than in the future...

If we implement everything incrementally by invoking YAGNI, the system can become increasingly difficult to modify, especially if every time we implement something new, we do it always in the simplest possible way

It is good to do the simplest thing that could work, however, we must remember to reapply that rule to keep the whole system as simple as possible. This means that we will refactor the code so that nothing violates OAOO.

This refactoring, by centralizing any functionality into a single class or method, leaves the system well positioned for the next change. Anything you need to do afterward, you can be almost sure that you can do it better by improving or replacing a specific and isolated point of your code.

Now let's see an example:

• I have to make a registration screen, which includes
  

  a <select> element that displays the countries of the world, so that when a person registers on my site, I can present them with content relevant to their country (or countries) of origin. As I'm going to apply YAGNI, I won't do more than this:

• Controller:

public List<Country> getCountriesList() {
    logger.info("Getting countries!");
    List<Country> countries = getSimpleJdbcTemplate().query(
        "select id, name from country", 
        new CountryMapper());
    return countries;
}

Following YAGNI, I don't have to do more. If this ends up being the only screen in my entire system that displays countries, I won't have wasted my time on complexity without benefits.

But then, it turns out that in another part of the system, on the screen to register orders, when I have to put in the address where I want the product I'm purchasing to be delivered, and among the information to indicate in the address, I have to include the country

It is then, and only then, -when I already have 2 conceptually separate places, and for which it is not convenient to use the same controller- that I add a service:

• Countries Service:

public List<Country> getListOfCountries() {
    logger.info("Getting countries!");
    List<Country> countries = getSimpleJdbcTemplate().query(
        "select id, name from country", 
        new CountryMapper());
    return countries;
}

• Controller: User Registration

public List<Country> getProductsToList() {
    return getCountryServices().getListOfCountries();
}

• Controller: Order Address Registration

public List<Country> getProductsToList() {
    return getCountryServices().getListOfCountries();
}

This is why when a system is finished, it gives the impression that "everything is built with layers from the beginning." But that's a mistake, a product of seeing the system as something static, as something that "was born" with the form it had at a moment in time, and not as something organic, whose structure changes as it adapts to the needs of the user who acquired it.

The architecture of the project is not something that is predefined before understanding the system. The architecture is an emergent structure, derived from the user stories that are gradually built as the system is shaped. If the system does not include among its stories the need to operate on multiple different databases (Oracle, SQL Server, etc.), then the DAO layer should never have to exist. If it doesn't add value, it's superfluous.

In the case of the previous example, we can go even further. Nowadays, most applications are RIAs built on HTML5 and JavaScript, where the logic in Java (or in C#, or in Groovy, or in Scala...) merely acts as a layer of data exchange and security between the persistence layer (typically a database) and the business logic and presentation layer (which are usually located entirely in JavaScript)

So why do we insist on using POJOs that don't add value? Why doesn't our code look like this:

public List<Map<String, ?>> getCountriesList() {
    logger.info("Getting countries!");
    List<Map<String, ?>> countries = getSimpleJdbcTemplate().queryForList("select id, name from country");
    return countries;
}

Why complicate our existence with one (or several) POJO (or DTO) if our system is so simple (at the Java level) that the methods exposed by our controllers only do CRUD and it is the JavaScript code that really brings the data to life? What's the point of working with POJOs, if our domain model is anemic?

Approaches like OData, sweep away in one fell swoop all this paraphernalia of making controllers to slowly and step by step build a query API that could be automatically derived from the data model. But OData is just a query mechanism, which comes to eliminate the need to keep "tying the cat"... that is, to make a custom layer of controllers each time.

Why don't Oracle, SQLServer, MySQL, PostgreSQL, etc. directly offer an OData API (or something similar) instead of forcing us to reinvent it each time?

It seems that our human traditions make us live in cycles within cycles:

1. We invented Cobol, to expose our data through procedures.
2. We invented relational databases (whose implementation we never completed) to allow access to data through query expressions (SQL).
3. We then invented stored procedures, to procedurally encapsulate our query expressions.
4. We then invented object-relational mappers, to achieve compatibility between multiple databases and be able to query them with a unified query language.
5. We then invented SOAP, to procedurally encapsulate our objects.
6. We discovered that in parallel, someone had invented REST, which allows us to query resources using query expressions represented by a URL.

And so... again and again, procedures, expressions, procedures, expressions, procedural approach, declarative approach... and every time we reinvent the wheel, we feel like we are breaking new ground... when in reality, we are just giving the wheel of fashion another turn.

Will we ever find the balance? (Or perhaps we have already found it... and just need to realize it...)

 Originally published in spanish in Javamexico

 

 

Architecture Rituals and Patterns: On the Importance of tying up the cat - Part 1

The Zen master and his disciples began their evening meditation. The cat that lived in the monastery made so much noise that it distracted the monks from their practice, so the master ordered that the cat be tied up during the entire evening practice. When the master died years later, the cat continued to be tied up during the meditation session. And when, eventually, the cat died, another cat was brought to the monastery and was tied up during the practice sessions. Centuries later, scholars descendants of the Zen master wrote treatises on the profound spiritual meaning of tying up a cat for meditation practice.

A few years ago, I attended an event organized by Sun to promote Java here in Mexico (I think it was called Java Dev Days). At that time, I entered a presentation, I think it was by Sang Shin (the creator of JavaPassion), in which he explained how simple it was to use JRuby with Ruby on Rails to create a "hello world" page. It was something like this video: http://www.youtube.com/watch?v=sas3KCKwyHI. It was the first time I saw RoR, so on that side, it seemed educational... however, most of the audience... was falling asleep!

That caught my attention... why were they all so bored, from what I had been able to observe the previous days, many of them were enterprise programmers, with the kind of clients that I have had, who give you very little information to work with, demand a lot, and want everything for yesterday..., so I started to listen to the comments that they exchanged quietly (or not so quietly) between yawns:

    Is that simple? It took like 15 minutes to make a hello world screen!
    How boring, I could do that with JSP in 1 minute
    Where does this guy get that this is simple? Did you see the mega directory structure that those commands created? That's not simple at all!
    You can tell this presenter spends his time giving courses, but he's never worked in real life, I'd like to see him with his RoR having to get the job done at 3 in the morning

At least the 2 rows sitting in front of me were not remotely impressed with RoR. And why would they be if in JSP you can make a hello world simply by putting something like this:

<HTML>
 <HEAD>
  <TITLE>Hello World</TITLE>
 </HEAD>
 <BODY>
  <H1><%= "Hello World!"  %> </H1></BODY>
</HTML>

Certainly, arguing that the 15 minutes of the multi-step video are simpler than this is absurd.

Well, some will say, (as part of me thought) the point of the example with RoR is not to show how simple it is to make a silly "Hello World" but to show the potential of RoR's MVC architecture to do much more complex things...

Let me see if I understood... are you telling me that I'm going to complicate my life by making something easy much more difficult and that's going to result in a simplification of the difficult? More complexity today == Less complexity tomorrow? That violates YAGNI. (Yagni means: You aren't going to need it)

And it's not that you have to avoid violating YAGNI just to follow tradition, as in the case of the cat in the monastery. Yagni has a reason for being: in systems development, change is the only constant, and if you get ahead and build something before you have a clear idea of whether you really need it, when you get to the point where you are really going to use it, it is very likely that what you really need will be very different from what you had predicted. This doesn't mean that your code shouldn't be flexible, but you have to be very careful, because you can end up with an over-engineered solution, ready for situations that will never occur. If you use too much energy today, fighting with the ghosts of things that will never come, when the real enemies finally arrive, you won't have the strength to face them.

Now, one of the big problems we have in learning programming is that when we are learning, we have to build examples like a Hello World in RoR to understand how it works. But as instructors, we have to be very careful in saying "look, with this example I'm going to show you how simple this is...". No. You don't make a hello world in RoR because it's simple. Let's not confuse our students. We don't go to the gym to sweat because it's less tiring than staying at home watching TV. We go because by doing so we will improve our skills, we go to prepare ourselves for future situations. So, in programming, not only Yagni rules, we also have another principle: OAOO. (OAOO means, once and only once).

OAOO is the root cause of the apparent complexity of RoR, and the archenemy of copy&paste. OAOO has other names, it is also known as the DRY principle (Do not repeat yourself). It's not good to have copies of the same code scattered throughout the project, because if there is a bug in one of them, and you didn't follow OAOO, you'll have to find and fix them all, instead of solving the problem in one place.

OAOO and YAGNI are like Yin and Yang. They balance each other. Without OAOO, YAGNI would slowly strangle us. With OAOO, you always have a system that you can improve (a phrase by Ron Jeffries).

OAOO alone would also kill us, leading us to an eternal refactoring of code searching for the holy grail of the generic code with the fewest possible lines without ever stopping...

We need both. We need balance.

How many times have you faced code like this in your work:

DAO:
    public List<Product> getProductList() {
        logger.info("Getting products!");
        List<Product> products = getSimpleJdbcTemplate().query(
                "select id, description, price from products",
                new ProductMapper());
        return products;
    }

Service:
    public List<Product> getListOfProducts() {
        logger.info("Getting products!");
        List<Product> products = getProductsDao().getProductList()
        return products;
    }


Controller:
    public List<Product> getProductsToList() {
        logger.info("Getting products!");
        List<Product> products = getProductsServices().getListOfProducts()
        return products;
    }

What value does this add? Why can't we simply do it like this (no Service layer, no DAOs layer):

Controller:
    public List<Product> getProductsToList() {
        logger.info("Getting products!");
        List<Product> products = getSimpleJdbcTemplate().query(
                "select id, description, price from products",
                new ProductMapper());
        return products;
    }

Why is it that if you don't tie up the cat before meditating... I mean, why is it that the principles of multi-layer architecture dictate that it should be done this way... really? Is that really the answer? NO!

If it doesn't satisfy an immediate need and doesn't help OAOO (obviously the redundancy in this doesn't help OAOO), we shouldn't do it. We are burning precious time and energy on "busy work" that nobody will thank us for.

Heresy! Many have told me when I tell them to stop having so many layers just because. Layers are NECESSARY. You can't remove them.

Of course, you can, and I'm not the only one who thinks so. JBoss Seam was designed precisely with the idea of eliminating so much unnecessary layering. And that was YEARS ago. Why hasn't it dawned on most people that making layers for the sake of making layers is not correct? Why do we still tie up the cat and buy a new cat every time we start a new project? Tradition.

Now, traditions are not all bad, (those cultures with traditions that don't help their survival simply become extinct). Where does the idea come from that putting so many layers helps in any way?

I will talk about that in my next post in this series...

(originally published in spanish on Javamexico)

Estimation: Performance Assumptions

A very common request (which often becomes a demand) made by clients is a performance guarantee, things like:

• I want all screens/pages to have a maximum response time of 3 seconds.
• I want all queries to have a maximum response time of 3 seconds.
• I want the system to run on my server (when we're lucky, they tell us the server's specifications).

What to do in such a case? When I had just graduated (more than a decade ago) my response used to be: "I need more data", "With the information I have, I can't guarantee that performance", "I don't even know what the algorithm for the indirect sales commission calculation process consists of, how am I going to guarantee that the screen where I see the result responds in 3 seconds???"

It almost goes without saying that my attitude was the wrong attitude...

My questions seemed logical to me, to give a guarantee of a page's response time of 3 seconds was like being told "you're going to transport a box of indeterminate weight, across an unknown distance, using an unknown mechanism and you have to guarantee that you can do it in 3 seconds".

Of course, my response was: If I don't know the weight, the distance, or the mechanism, I can't guarantee the time....

And the inevitable response from my bosses: Don't tell me why not, tell me "what would have to be true to make it happen"... was irritating to my ears.

But they were right, if you're asked for a guaranteed transport time, and you don't know the weight, the distance, or the mechanism to use, what is expected is not that you say "I lack that data", what is expected is that you say under what conditions you could do it in the specified time (if later the client cannot provide them, then it's not your failure).

Of course, this doesn't mean you shouldn't ask. It's always best to ask. But paradoxically, you should ask, but not worry too much whether or not you get an answer. There will be clients who are interested in you building the right thing. Those clients will try to answer your question by giving you the parameters you require. There will be other clients for whom the "right" thing doesn't matter, the only thing that matters to them is that it's done "correctly" (that it's well done, even if it's not useful to them). This second type of clients will often be irritated by your questions, they might even answer "you're the expert, I don't know about this, you tell me what you need to do it in the time I require"... There will be cases where your client is lucky, and the right thing and the thing done correctly will coincide... but always try to do the right thing, but if your recommendations are ignored, don't get irritated, and switch modes. Don't think that your client is "stupid" or "bad" for demanding the "what needs to be true". Ultimately, they are right, you are supposed to be the expert: tell them "what needs to be true", and let them decide whether or not they have the budget.

Thus, all the missing parameters must be assumed. Let's take the most extreme case, they ask for a response time of 3 seconds, and they give you no other clues... and there are various requirements in the system (such as "display commissions for indirect sales"). You don't know how many sales they have, but you know it's a multinational corporation, and they are certainly not "a few."

What to do? Let's go back to the example of transporting a package:

Weight: Unknown

Distance: Unknown

Mechanism: Unknown

Transport time: 1 hour.

What would you do?

Well, you could, for example, say: In the basket of my bicycle, fits 1 liter of water. And in one hour, on my bicycle, I cover let's say 40 kilometers, then:

Weight: 1 kilo

Dimensions: maximum 15 cm square (the volume that fits in the basket of the bicycle)

Distance: maximum 40 km

Mechanism: Bicycle

Transport time: 1 hour.

Now, the big mistake why assumptions like these are considered the mother of all evils is because the one who assumes them keeps them until the day the client wants to send a package, and it turns out that half a ton of bricks needs to be sent 100km away in a maximum of 2 hours... and there's no way in the universe to do that in the basket of your bike...

And then, comes the typical sermon: "don't assume!" "ask!" (Which becomes particularly annoying because you clearly remember asking the client and they never gave you a clear answer)

Was the error in the assumption? NO.

The error was that you did not make your assumption PUBLIC. You didn't verbally discuss it with the client, you sent it by email and tried to get them to sign a document accepting it. Now, of course, some might think: "I would have to list all the possibilities" — what if the package is 100kg, what if it's 10 square meters, what if I have to deliver it in 15 minutes? That's the wrong approach. The correct approach is to give one example, but very clearly defined.

Of course, some will say, there will be something you overlook: what if the user asks to send 1 kilo of uranium? It meets all the criteria, but the radioactivity will kill you! There's no way to think of everything.

Indeed, there isn't. But that's no excuse not to try to define things as precisely as possible. If you strive to define your assumptions as best you can, they will gradually become more complete, and after a few years, experience will allow you to include enough factors so that the possibility of being asked to transport something that violates your assumptions and causes you harm is very remote. If at the beginning of your career as a developer, 9 out of 10 assumptions lead you into problematic situations, and 10 years later, only 5 out of 10 do, that is a success.

Returning to the scenario that interests us as developers, if you are asked: “I want all queries to have a maximum response time of 3 seconds”

Establish assumptions:

• Maximum number of records per query.
• Maximum data weight of the query in Megabytes.
• Network speed (Mbits/s).
• Hardware characteristics of all servers and client workstations involved.
• Type of operation (if it's in SQL, maximum number of records, presence of indexes, maximum number of records in the tables, maximum number of joins, version of the pseudo-RDBMS, etc., etc., etc.).

In civil engineering, if an engineer wants to know the strength of a steel bar 10 cm thick and 10 meters long, they can look it up in a catalog...

Isn’t it time we built that kind of knowledge for ourselves?

As much as I've looked for tables with the above data, where someone has tested certain "typical" hardware/software configurations and basic algorithms”, something like:

If you perform a Join between 2 tables with such and such columns in Sql Server 2008R2, on an i5 server, with 8GB of RAM and a disk of certain characteristics, on a network of... and its combinations (after all, we are programmers, we create algorithms, let's make one that generates X combinations).

Will it be a benchmark that works absolutely? Of course not, but that's not the goal, the goal is to be able to offer a reasonable assumption, and give your client the confidence that if the conditions you set are met, you can guarantee a result.

Then, if something else happens along the way... well, it's not your fault, you couldn't control it, and the number of clients who will understand that you didn't meet the goal because they (or the circumstances) violated your assumptions is (at least in my experience) much greater than those who would understand if you tell them from the beginning "why not" instead of "as if."

Remember, assumptions are not bad. If you ask a civil engineer to build your house, you don't expect to answer their questions about the strength of materials, that's not your job. The job of the consultant (whether a software developer or a civil engineer) is to find the conditions under which "it can be done," and in software, fortunately... or unfortunately, almost everything can be done, if you have the right assumptions.

 Originally published in Javamexico

Estimation: Assuming is good, we must assume as much as possible

 

Before continuing with more concrete examples of assumptions, I find it important to tell you that when I started in software development, I often encountered this situation:

Project Leader: The client said that feature XXX doesn't seem useful to them. They don't understand why we made it this way when they needed something else. 

Programmer: Well, I assumed that... 

Project Leader: Well, next time don't assume: ask! Programmer: Sorry, it won't happen again.

And a few days later:

Programmer: How should we make feature YYY? 

Client: Well, in such a way that we maximize efficiency and add value to the business... 

Programmer: Yes, but, is the relationship between products and orders one-to-many or many-to-many? Client: (What is this guy talking about?) Sorry, I have another meeting, we'll look at it later. 

Programmer: But but...

And finally:

Project Leader: The client said that feature YYY doesn't seem useful to them. They don't understand why we let them down again, they needed something else. 

Programmer: I did ask, really, but he only explained it very superficially. 

Project Leader: Well, next time, get detailed information. Programmer: Sorry, it won't happen again.

And so, always caught in the same loop, without control, sometimes doing well, sometimes not, depending on how much the programmer's intuition matched the client's reaction to the tested software.

Sadly, what the programmer learns after several such cycles is that assuming is bad. Wrong conclusion. Assuming is the best thing we can do at the start of a software project. Let's assume everything we can possibly assume. But let's be explicit about our assumptions.

Let's change the dialogue, negotiate differently, so it looks like this:

Programmer: How should we make feature ZZZ? 

Client: Well, in such a way that we maximize efficiency and add value to the business... 

Programmer: Here I have (many detailed and explicit assumptions) a prototype of the screen, here is my user story and I've already written some acceptance criteria, please give them your approval so I can start programming. 

Client: Ok, I'll review them later and let you know. 

Programmer: Perfect!

Of course, one of the most common problems we might encounter is that the client finally doesn't have time to review and never gives us approval or does give it, but without really reviewing the assumptions.

Project Leader: The client said that feature ZZZ doesn't seem useful to them. They don't understand why we made it this way if they needed something else. 

Programmer: Well, I delivered a prototype, the user story, and the acceptance criteria, and he gave approval, if you want we can review them to see if they correspond, but the Tester already has evidence that there is correspondence and all the bugs have been fixed. 

Project Leader: ... Well, we'll have to tell the client that if they want it different, they need to request a change.

This is the point we want to reach. Yes, the best we can do, the ideal that the agile movement aspires to, is to build what the client really needs, but often it is not achieved on the first attempt, because the first who doesn't clearly know what they need: is the client. And the technique to get there is assumptions. But explicit, visible, clear assumptions, if they are about screens, with prototypes, if they are batch processes, with examples of congruent input and output data.

The bad thing is not the assumptions, the bad thing is when we assume "in secret", without evidence, without giving visibility.

In the next part of this series, we will continue exploring more vaguely defined assumptions and how we can turn them into refutable, bounded assumptions, which, when violated (by the client), result in more work and paid time (remember, the goal is not to finish the project, software is like a plant, like a living being, the goal is to cultivate it, let it grow, mature, adding value to the client if possible, indefinitely).

We must strive to do the right thing, without forgetting to do it correctly.

Originally published in Javamexico

Estimation: Negotiation and the difference between doing the right thing vs. doing it right

In English, there's a saying: "There is a difference between doing the right thing and doing the thing right."

In software estimation, and the negotiation process necessary for a consultancy to build software for a client, the difference between one and the other is tremendously important.

To "do the right thing," we must clearly understand what the client really needs (regardless of what they ask for), whereas to "do it right," it's not relevant whether what we do will actually be useful to the client or not; what's important is whether what we did is well-built.

An extreme example:

A client comes to you and describes a single-passenger transport vehicle that allows them to travel anywhere. You build them a bicycle, but they were thinking of using the vehicle in Alaska, in the snow—they were looking for a snowmobile.

Ultimately, it doesn't matter how well you built the bicycle (done correctly) because it's not useful to the client since what they needed (doing the right thing) was for you to build them a snowmobile.

During the negotiation process carried out during the construction of software, the difference between "doing the right thing" and "doing it correctly" has an important peculiarity: Who has the information required to achieve one objective or the other changes drastically.

For "doing it correctly," all the knowledge is with the consultancy that builds the software, in the technicians, the developers, the designers, who, if they are good, will make the most usable screens, and will use the best architecture and algorithms, strictly adhering to a process that helps ensure a well-built product.

Conversely, for "doing the right thing," the information is with the client. Only they can, although often not at the beginning of the software development, determine if what is being built for them is ultimately what they need. Yes, and only if the client is willing to go through this process of self-discovery, in which their assumptions will be carefully examined, can they finally conclude whether that nebulous idea in their head is really what they need, or if instead, they should have asked for something else from the beginning.

Of course, the consultancy, with its analysts, developers, architects, testers, etc., must seek to facilitate dialogue with the client and help them quickly enter the cycle of assuming, implementing, validating, and re-assuming until they construct "the right thing," but it's important to remember that ultimately the information comes from the client, not the consultancy, and if the client, due to some problem of their organizational culture, is not willing to invest the time required to clearly define "the right thing," the consultancy will not be able to build it (you can lead a horse to water, but you can't make it drink).

There is a prayer that says "God, grant me the serenity to accept the things I cannot change, the courage to change the things I can change, and the wisdom to know the difference." Well, in this case, we need to be very clear about the difference: Only the client has the power to help us build "the right thing," the consultancy is just a facilitator, it's very important to remember this because when we are constructing the proposal, we must ensure it helps to build "the right thing," but at the same time, our responsibility must not go beyond "doing it correctly."

I have had to deal with multinationals, who give me just a few hours to prepare a proposal for months of development. The amount of effort wasted in such developments is absurd, and yet, it was useless for me to insist that 3 to 6 hours are not enough to estimate a project of 3 or 6 months duration, too much remains unspecified. At first, when I faced this situation, my attitude was "it can't be done," "avoid assuming," "ask the client for the missing information, or if not, do nothing." Sounds very nice from an abstract perspective, but that attitude doesn't pay the bills... so I changed it.

I still tell the client whenever I can that the time they are giving me is not sufficient, and that it is very risky to do a project with so little time to plan, but once the client indicates that my warnings are falling on deaf ears, I immediately change strategy: This client isn't interested in me "building the right thing," what they care about is having something built that "comes as close as possible to that," and I'd better "do it correctly," because in the end, if the approximation isn't close enough to what the client realizes they needed to receive (minutes after receiving it), recriminations and attempts at penalties will follow.

Fortunately, "doing the right thing" is subjective, whereas "doing it correctly" is completely objective. If you needed a snowmobile, and I built you a bicycle, and the very limited assumptions of the proposal are met by the bicycle, there will be no way for you to penalize me, or refuse to pay me, and in the event of a lawsuit, you will lose, because I will be fulfilling the contract.

The Agile Manifesto says "Customer collaboration over contract negotiation," which is another way of saying "prefer doing the right thing over doing it correctly," but the Agile Manifesto clarifies: "while there is value in the items on the right, we value the items on the left more." It's the same with "doing the right thing" vs. "doing it correctly," both are valuable, and certainly the best would be to do both, "the right thing, correctly," but we must have the wisdom to know the difference between what we can change... what is in our hands, and what is a shared effort, and that if the other party outright refuses to cooperate, it will not be possible to achieve.

It is very important to do this with a cool head and an enthusiastic and cheerful attitude. If a client is difficult in the first place, and we also treat them rudely, we will get nowhere; the warmth of kindness has more impact than the coldness of aggressiveness. As much as possible, we should seek common ground, empathize with the client, and move forward together, but we must also have clear boundaries drawn so that if things get out of control, we can cut cleanly, without irreparable losses to the continuity of our business, otherwise, we'd fall into Stockholm syndrome...

Don't miss the upcoming episodes of this exciting series on estimation, where we'll see different examples of real-life situations you might face, and how you can emerge victorious from them...

Update: I came across this article on InfoQ, which I see as very related to the topic I'm discussing here and I loved this example, so I take the opportunity to share it with you:

People buy a hammer to drive a nail to be able to hang a picture - they know they can achieve their goal (hang the picture) with the acquisition of the hammer.

Unfortunately, in the context of software development, things are not so straightforward between what is delivered (the software) and the business objective that is desired to be achieved. Many people do not even try to reach the business objective. This creates a significant risk that the provider only delivers what the client asked for - software that satisfies a vague set of requirements - instead of what the client really needs, which is to achieve their business objective.

Unfortunately, it is the same clients who tell the provider: Do exactly what I asked, stick to the contract, do it correctly, I have already decided that this was the right thing... without realizing the harm they are doing to themselves. As providers, we have the responsibility to ask: Why do you want me to make you a hammer? But if the client ultimately refuses to explain the purpose, we must accept that the client is ultimately responsible for their own well-being, and make their hammer, even if their plan is to hit themselves with it... imagine if knife manufacturers spent their time stressing about "what if someone cuts a finger," or stove manufacturers about "what if someone gets burned," practically all industries would end, since there is no limit to the misuse that can be given to things...

Clients are not children, and while we must clearly warn them of the consequences of their actions, the responsibility for their actions lies with them.

In the next part of this series, I will talk about techniques for dealing with performance requests (or demands) in situations where apparently there is not enough contextual information.

Originally published in Javamexico

Estimation: Is assuming the mother of all estimation errors? Or perhaps it is "to suppose"? Or maybe it is just our own ignorance?

I recently read a tweet that said:

Congratulations to "Assuming" for being the MOTHER OF ALL ESTIMATION ERRORS in software development!!!!

— SoftwareEvangelist (@vanessa_amaya) May 10, 2013

And I wondered: does the author realize... that she is assuming that assuming is the problem?

First, let's review the meaning of the word:

assume: assume. (From Lat. assumere).

1. tr. To take to oneself, to take for oneself.
2. tr. To take charge of something, to be responsible for it, to accept it.
3. tr. To acquire, to take on a greater form.

In software, the meaning we commonly use based on my experience is "To take charge, to be responsible for something, to accept it." What do we take responsibility for? The assumptions we use to build our estimate. Should we instead seek to be irresponsible? In my opinion, the problem is rather the assumptions themselves. If the assumptions do not align with reality, we end up delivering things poorly, for example in the scenario Ezamudio discussed in the previous post of this series, the problem escalated because the technician assumed he had chosen the correct part; in software development, it is more complicated, as there are several intertwined assumptions. But returning to the beginning, is assuming really the mother of all problems? No. Is supposing then the mother of all problems? I don't agree with that either, let's review the meaning of suppose:

suppose. (From Lat. supponere).

1. tr. To take for granted and existing something.
2. tr. To pretend, to give an ideal existence to what really does not have it.
3. tr. To entail, import. The new acquisition he has made implies excessive conservation expenses.
4. tr. To conjecture, to calculate something through the indications one has.
5. intr. To have representation or authority in a republic or in a community.

The meaning we commonly use in software, (at least those of us involved in its construction) is to conjecture, to calculate something through the indications one has, while we consider that the client sees it more as "To take for granted and existing something." Contradictory? Superficially, one might think so, but if examined more deeply, there is no contradiction. Supposing is a necessary something, we get out of bed, and we suppose we are awake, we drink water from the jar and suppose it is not toxic, we go out and drive to work and suppose it will be relatively safe. We do not have absolute certainty, but we also do not torment ourselves, we act taking it for granted, and if in the end the water or food causes us indigestion, we deal with it, if we start having indigestion several days in a row (if it does not kill us) we will change our diet and the water we drink. The same applies to software assumptions.

Supposing is a necessary activity in the scientific method (step 3):

1. Observation: Observing is applying the senses attentively to an object or a phenomenon, to study them as they really present themselves, which may be occasional or causal.
2. Induction: The action and effect of extracting, from certain observations or particular experiences, the particular principle of each one.
3. Hypothesis: Formulation through observation following the norms established by the scientific method. <---- ASSUMPTION!
4. Test the hypothesis by experimentation.
5. Demonstration or refutation (antithesis) of the hypothesis.
6. Thesis or scientific theory (conclusions).

Basically, we Observe, Analyze, Assume, and Test. If it works, we keep the assumption, if not, we repeat until we find the "correct" assumption. Now, this is the most important thing about the scientific method: We never reach the correct, the path to truth is like the limit in infinitesimal calculus, we are getting closer each time, but we never arrive.

The most important lesson of the scientific method is implicit: It's a Sand Mandala. Assumptions are made, only to be replaced by better ones. Thus, in software, assuming our assumptions are true is not a mistake, it's part of the method; the mistake comes when we forget that we are just at step 3, step 4 is still to come, where, without neglecting our responsibility, we test if our assumptions match reality. Therefore, assuming and supposing are not the mother of all problems, the mother is rather the lack of experimentation, and this lack of experimentation stems from treating assumptions incorrectly.

The first rule to follow for assumptions, again a rule from the scientific method, is that the assumption must be refutable (principle of falsifiability), what does this mean? There must be a concrete way, through a particular observation, to declare the assumption (provisionally) true or false. (An rrefutable assumption would be one that requires an exhaustive search of all possibilities to refute it). For example, I've been given RFPs to make systems that say "the system must implement all the relevant business rules for the business process to be automated" and then, nothing, no list of which rules are. Demonstrating whether we are meeting or not meeting that assumption cannot be refuted, the list of rules is potentially infinite (and clients certainly take advantage of that). What can we do?

When I was younger, what I used to do was demand the list of rules, or say that I couldn't do anything and refuse to estimate the time that would be necessary. Needless to say, that attitude only got me into trouble with my bosses, lost business, and dissatisfied clients. Although that's the path indicated in estimation books, it's not the correct path in the real world.

Of course, there are clients who are willing to give you the list of rules, so when you're in this situation, don't fail to ask for the list, but if they can't (or won't) give it to you, don't suffer, accept it, and continue... but cautiously. A few months ago, I watched a nascent consultancy suffer a horrible death and enter zombie mode: they sold a system with open-ended assumptions, according to them, it was going to be finished in 2 weeks; when I met them, they had been working for free for the government for a year, under threat of a lawsuit for breach of contract if they were to "crack" and not finish implementing all the rules that the client might think of. I couldn't help them anymore, what they needed was a good lawyer. How then to avoid falling into that hole?

Some would say: Simply reject the business. Yes, it sounds nice when you have a lot of money, have no responsibilities, and if you don't receive income you're the only one who doesn't eat. But what if your family depends on you? Can you really afford to reject the work? Of course not. Don't reject it then, nor tire yourself explaining why you can't "implement all the relevant business rules for the business process to be automated", the right answer is: define the scope yourself.

Place an assumption in your proposal in the following way: "It is assumed that these business rules will not exceed 10. It is assumed that the list of all business rules to be implemented will be received X days after the project has started. The rules must be defined as algorithmic step-by-step procedures, each consisting of no more than 10 CFP (Cosmic Function Points according to the standard ISO/IEC 14143/1:2003). It is assumed that all the information to be used by the business rules is that which is fed into the system through the user stories defined in the scope section of this proposal. If the list of business rules is not received at that time, or if a greater number of rules are received, the scope change procedure described in this document will be executed, which may have an impact on the project times and costs." There, you've defined the scope.

By doing this, essentially you are answering the question that some clients would surely ask you when you told them that you couldn't "implement all the relevant business rules for the business process to be automated". Some clients would ask: Why not? And you would tell them: Because I don't know how many there are, nor how complex they are, and you're only giving me 3 months to do them. Well, the client would say: How many can you do in 3 months?

And there is where, when I was younger, I would clash most strongly with reality: I simply couldn't answer that question! I could only say "because not", when the right answer is to say "as if". This is not a silver bullet, there will be clients who are not willing to accept a defined scope. In those cases, if the client is not willing to pay for time and materials, then yes, the right answer is thanks, but no thanks, I can't do the project.

In the next part of this series, we will delve into the benefits of assuming, and why the myth has been generated that one should not assume.

 Originally published in Javamexico

Estimation: Why do we do it?

 

There are those who consider that estimating is impossible and a waste of time, a useless endeavor. There are those who believe it can be done, but only under certain conditions, and there are those who think that the secret lies in following a certain method... However, before discussing these points of view, I want to focus on a question often omitted in articles and books on estimation:

Why do we estimate? And I'm not talking about the theoretical reasons typically used in books on the subject, but rather, out there, in the real world. I don't want to generalize, so what I will say next is strictly based on my own experience.

We don't estimate to know how long the project will take; the client usually has already set a deadline that is unlikely to change.

Why do we estimate then? We estimate to see if we can do something within the time and budget that are already established and that sounds to the client like what they asked for (because if one thing is certain, it's that most clients don't really know what they want until they've seen a couple of progress demos).

The agile community tells us: don't estimate, do everything by time and materials, although in reality what it's saying is "don't estimate beyond the next sprint" and "lose any client who is only willing to buy projects at a fixed cost and time." It would be nice if the world were that easy, but the fact is that it's not: most of the clients I've encountered in the consulting world want a budget upfront... Those of us who would like everyone to understand Scrum sometimes find it absurd, a display of their ignorance some say, but let's put ourselves in their shoes: who, when taking their car to the shop, expects to be told: we're going to fix your car iteratively, give us your bank card with an open voucher and then check your balance to see how much it was at the end...

And yet we go to the doctor, and while each visit may have a fixed cost, we don't ask the doctor to give us the cost of the medicines before diagnosing us, nor do we force them to tell us the maximum number of consultations we'll need... Medical treatment is completely iterative... Maybe one day we'll recognize the similarities between curing the most sophisticated biological system we know and creating or composing new systems... In the meantime, let's think... Why are there fixed-cost projects that do work? The level of uncertainty clearly visible to the educated mind at the start of the project should guarantee that this never happens... What's different in those projects? Maybe it's not obvious, but in my experience, it's quite simple: the scope was better defined, either because the client themselves had already done some pre-analysis, or because the consulting team did the same... SACRILEGE! Am I daring to say that the correct way to carry out a project is with Waterfall?

NO! To believe that is to ignore the principle of diminishing returns: those who believe that this indicates Waterfall is the answer are making the mistake of thinking that if some analysis is better than none, then double or quadruple the analysis will bring a benefit directly proportional. Well, welcome to the real world: things don't work like that, if you do a lot of analysis, you also consume a lot of time and while your understanding of the situation becomes more precise, it also becomes progressively outdated... The world doesn't stop changing because you analyze a particular point of it in time and by the time you turn to compare your analysis with the current reality, the world has already changed: the conclusions of your analysis are now obsolete.

Zero analysis? Bad. Exhaustive analysis? Bad... What to do then? Find out how much is "enough" but first understand enough for what? Many books on estimation talk about achieving 75% accuracy, as the ultimate goal of a well-done estimate. In my experience, that perception is wrong. What good does it tell a client: to succeed in what you're asking, I need one million pesos, if the client only has (or claims to have) a third of that? It's no use, the client won't have their product, and we won't have our project. What to do then?

Look for the "as if," the first time I heard that phrase it seemed naive: do a one million project for a third? Impossible! But then it happened that some people asked me: why? And I complicated my life giving a thousand explanations about the complexities that are part of software development (many of which I didn't understand then... and many of which I still don't fully understand now) and... I couldn't convince them, and ended up trapped in horrendous Dead March projects from which I not only received pressure from the client, desperate because the project was nowhere near completion by the deadline, but also from the company I worked for, equally desperate as the project's margin eroded away...

And after finishing that project, we embarked on another just like it... How do we break that cycle? Is  it true, as I recently read on Twitter, that assuming is the mother of all problems in system development. I will talk about that in my next post...

 Originally published in Javamexico

SQL 2012 Bug: NVarchar changes where evaluation order

Today I arrived late at my house because my team had a big problem migrating a stored procedure from SQL Server 2000 to SQL Server 2012. Countless hours lost trying to find out what we could be have been doing wrong...

On the end, in turned out, all the trouble was because of a bug in SQL Server 2012.

Here is what we found, lets say you have a table "T_1":

CREATE TABLE [dbo].[T_1](
[C] [nvarchar](50) NULL
)

Now, lets say you add some rows to it.

INSERT INTO [dbo].[T_1] ([C]) VALUES ('P')
INSERT INTO [dbo].[T_1] ([C]) VALUES ('Q')
INSERT INTO [dbo].[T_1] ([C]) VALUES ('R')


Now write this query:

select * from T_1 where ISNUMERIC(C)=1 and CONVERT(float,C)=0.0

And you will get a nice error message:


Msg 8114, Level 16, State 5, Line 1
Error converting data type nvarchar to float.

¿What is the bug? You shouldn't be getting an error message! the "Error converting data type nvarchar to float." is generated by the CONVERT(float,C)=0, but SQL should never run that code because ISNUMERIC(C)=1 evaluates to false, and if the first part of an "and" is false, there is no point in executing the second part, the result is going to be false anyway.


Well you might say, maybe SQLServer has always done things this way... well, no, it has not. In SQL 2000, that query executes correctly!

And in SQL 2012, there is a workaround, just change from nvarchar to varchar:


CREATE TABLE [dbo].[T_2](
[C] [varchar](50) NULL
)


INSERT INTO [dbo].[T_2] ([C]) VALUES ('P')
INSERT INTO [dbo].[T_2] ([C]) VALUES ('Q')
INSERT INTO [dbo].[T_2] ([C]) VALUES ('R')



Now if we write (note we are now working with the table T_2 that uses varchar) :

select * from T_1 where ISNUMERIC(C)=1 and CONVERT(float,C)=0.0

We will get no error.

Why is this happening then?

It seems to be a bug in SQL Server 2012 execution plan:

 

 

As you can see in the image, SQL 2012 inverts the predicate when working with nvarchar!

On the other hand, for T_2  the table with a varchar column:

As you can see, here the the order of evaluation is preserved, and things work like they should.

Now what can we do if we are not allowed to change the type of the column in the table?

I tried using a CTE, but it does not work, the execution plan is the same faulty one:

with V_1 as (select * from T_1 where ISNUMERIC(C)=1)
select * from V_1 where CONVERT(float,C)=0.0

Using a subquery also fails:

select * from (select C  from T_1 where ISNUMERIC(C)=1) V_1 where CONVERT(float,C)=0.0

We can change the type explicitly to varchar in the query, that fixes the problem:

select * from T_1 where ISNUMERIC(C)=1 and CONVERT(float,convert(varchar(100),C))=0.0

But,  what if we actually have an Unicode string with chars that will get damaged by a conversion to varchar?

I shouldn’t be a problem…. What do you think? any other workaround?

UPDATE: I have submitted this bug to Microsoft Connect, click here to see my bug report

Web Slices: you need to use Alternative Display Source or javascript will not work

If your WebSlices use the Basic Web Slice model, javascript will not work (this file is WebSlice.html):

<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server">
    <title></title>
</head>
<body>

<div class="hslice" id="SliceID">
     <span class="entry-title">Title of the web slice</span>
     <div class="entry-content">Preview of the <a href="#" onclick="document.getElementById('Message').innerHTML='Hello'; return false;">web</a> slice
       <div id="Message"></div>
     </div>
     <p>
     Hola Rebanadas Web!
     </p>
  </div>

</body>
</html>

you have to use Alternative Display Source to make javascript work (this file is WebSlice.html):

<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server">
    <title></title>
</head>
<body>

<div class="hslice" id="SliceID">
      <span class="entry-title">Title of the web slice</span>
      <a rel="entry-content" href="AlternativeDisplay.html" style="display:none;">Alternative Display Source</a>
      <p>
      Hola Rebanadas Web!
      </p>
   </div></body>
</html>

 

and in the file AlternativeDisplay.html you put the code that use to be in WebSlice.html (and that needs to use javascript):

<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server">
    <title></title>
</head>
<body>
     <div class="entry-content">Preview of the <a href="#" onclick="document.getElementById('Message').innerHTML='Hello'; return false;">web</a> slice
        <div id="Message"></div>
      </div>

</body>
</html>

How to migrate your local user profile to the domain

Apparently, there is no easy way (this or this do NOT work), there use to be a tool to do this (Moveuser.exe), but it stopped working with Windows Vista.

The way to do it now is to write a VBScript that uses WMI, happily, I have found someone that has already done it here.

It basically seems to be using the ChangeOwner Method of the Win32_UserProfile Class, one problem I have found so far is that if the profile to be “moved” in to the domain is heavy (many Gbytes of weight) the ChangeOwner method can take a really long time to do its job. A possible solution then is to move the files outside of the profile dir, run the migration process, and copy the files back (you might need to change the permission information of those files to be able to copy them back)

How to move C:\Users to D:\Users

There is no user friendly way to move your C:\Users folder to D:\ but the user unfriendly way to do it is not too hard (it was a little hard to discover, I learned how to do this when I decided that I wanted to have C:\Winnt\Profiles dir at a different location, after I lost most of my data when my main disk failed, back when Windows NT 4.0 was the latest and greatest OS from Microsoft)

Basically, after you installed Windows (this instructions are for Windows 7 but the general approach is valid since Windows NT 4.0)  you have to:

1. Login with “WhatEverIsYourUserName” account.
2. Run the Command Prompt (cmd.exe) as an Administrator
3. 
4. Copy the C:\Users\ folder to D:\Users using Robocopy
   

   robocopy C:\Users D:\Users /E /COPYALL /R:0 /Z /XJ

   
   



5. Open regedit and modify the ProfileList entry so that it looks like this:

1. Create a new (Administrator) Windows account that you will use to test the configuration change
   
   
    



2. Restart



3. Login with the “Test” Windows account



4. Delete C:\Users



5. See that the  Start menu still works (if you made a mistake, your Start menu is now empty)



6. Delete the registry Key for your “WhatEverIsYourUserName” account (if you still want to use it). (The registry Key is the yellow folder with the S-1-5… name that contains the value ProfileImagePath that points your now obsolete C:\ folder, remember,  you have to delete the whole folder/key, not just the ProfileImagePath value):
   
   
    



7. Log out



8. Now log in with your “WhatEverIsYourUserName” account



9. If you take a look at the registry now, it will look like this (The value in ProfileImagePath now starts with “D:\” instead of “C:\”):
   
    



10. Now you can delete the test account.

And that is it, now the users accounts live in D:\Users\

Recover sa account

Today, a friend from the office locked himself out of SqlServer 2008 R2 Express, somehow he managed to remove the windows Administrator account form the list of sysadmin accounts in the SqlServer express installed in a development server.

I thought there was no way to recover from a mistake a like that, but another friend found the way and gave us a link with the solution.

Basically, we need to stop the SqlService, and restart it in "single user mode" and "minimal config mode"

sqlservr.exe -m -r -s SQLEXPRESS

And the open another command window and run this commands:

osql -E -S .\SQLEXPRESS 
exec sp_password @new='changeme', @loginame='sa' 
go 
alter login sa enable 
go 
exit 

And that was it. Remember, without the special "-m -r" options, the osql commands will fail.

NOTE: In case your SQL Server is not configured to use Mixed Mode for authentication, you may also need to make some modification to the registry settings.

Basically you need to find the registry entry for the instance:

Default instance:

HKLM\Software\Microsoft\MSSqlserver\MSSqlServer\LoginMode

Named instance:

HKLM\Software\Microsoft\Microsoft SQL Server\Instance Name\MSSQLServer\LoginMode

And  change the value of LoginMode to 2.

Formulating Expressions a Step at a Time: Lazy Evaluation

I am reading this book, and I found the section with the title “Formulating Expressions a Step at a Time” particularly interesting:

First, the book shows you this the query for the sentence "Get pairs of supplier numbers such that the suppliers concerned are collocated (i.e., are in the same city)" written in Tutorial D:

( ( ( S RENAME ( SNO AS SA ) ) { SA , CITY } JOIN
   ( S RENAME ( SNO AS SB ) ) { SB , CITY } )
       WHERE SA < SB ) { SA , SB }

And then it proceeds to show you how to write this query in a more readable (step by step) way:

WITH ( S RENAME ( SNO AS SA ) ) { SA , CITY } AS R1 ,
    ( S RENAME ( SNO AS SB ) ) { SB , CITY } AS R2 ,
    R1 JOIN R2 AS R3 ,
    R3 WHERE SA < SB AS R4 :
    R4 { SA, SB }

Finally, it shows you how to write this query in SQL:

WITH T1 AS ( SELECT SNO AS SA , CITY
            FROM   S ) ,
    T2 AS ( SELECT SNO AS SB , CITY
            FROM   S ) ,
    T3 AS ( SELECT *
            FROM   T1 NATURAL JOIN T2 ) ,
    T4 AS ( SELECT *
            FROM   T3
            WHERE  SA < SB )
    SELECT SA , SB
    FROM   T4

Thanks to the “with” keyword, both in SQL and in Tutorial D, it is possible to deal with this query in a “step by step” way, instead of having to deal with it a single hard to write and hard to read expression (note that this is not a recursive query, so the with keyword is not being used for that in this examples)

Sadly, so far I have been unable to find a equivalent for this syntax in Dataphor… While it is possible to write something like (I am not 100% confident the syntax is right, but I think it should give you the general idea):

var R1 := S {SNO SA}
var R2 := S {SNO SB}
var R3 := R2 JOIN R3
var R4 := R3 WHERE SA < SB
select R4 {SA, SB}

In Dataphor, the variables (R1, R2, etc) are not lazily evaluated, and therefore the performance is not as good, as in, for example, the SQL case (I ran a similar example in SqlServer, and the expressions were evaluated lazily, at the end, instead of one by one, resulting in far better performance). Maybe I am doing something wrong?

I wonder how hard would it be to make Dataphor generate SQL using the WITH keyword with those databases that support it (so far the latest versions of SQLServer (2008) and Oracle (10 & 11) seem to support this syntax)… I guess it is time to ask the Dataphor authors…

Null versus None

None means Nothing, Nothing is a concept that describes the absence of anything at all. Nothing is sometimes confused with Null, but they are very different concepts because Nothing means absence of anything, while Null means unknown (you do not know if there is a thing or not).

For Nothing, the normal Two valued logic applies (Nothing=Nothing : true, Nothing = Something : false), for Null, Three valued logic is necessary(Null=Null:unknown, Null=Something:unknown). Unfortunately, this 2 concepts have been used interchangeably without much thought, to point where the most common use for Null in relational databases is to mean Nothing (even when Null was designed to represent unknown by Codd). This confusion is aggravated by the fact that many mainstream application languages (Java, C#, C, etc) use the null keyword to mean uninitialized variable which easily maps to the interpretation that null means the variable is pointing to "nothing" (no object).

But for databases, Null was not invented to represent nothingness, was invented to represent that the value of something was not known (maybe be something, maybe nothing, we just do not know).

Now that Chris Date wants Null to be removed from Relational Databases, so that the incongruence and confusion brought in by Three valued logic is eliminated, the developers, accustomed to use Null to represent Nothing, resist to the idea asking: How am I going to represent the fact that a Person is not married? I use to do that by marking the Marriage Date? as a nullable Date. Now what? I need to split the table into 2 tables just to represent the fact that the Marriage Date? is not mandatory? That of course seems like the obvious, elegant (if extremely cumbersome answer). But the practical developer refuses to get into that trouble, it is just too much effort, it is simple easier to continue using Null. But... what about Nothing? why not just simply add "Nothing" as a possible value for to the Date domain? That way it is possible to say that the Person has no Marriage Date?, and still stay inside the realm of two valued logic.

Is this solution, in any way in conflict with The Third Manifesto? I really would like to know… I wonder what will be the opinion of the community in the  C2 Wiki

Find me a circle

A circle is a simple shape of Euclidean geometry consisting of those points in a plane which are equidistant from a given point called the center. The common distance of the points of a circle from its center is called its radius.

But Circles can not, and do not exist in reality, anything that seems to be a Circle is actually a polygon with a very high number of sides... the closest to a Circle that we can get is to have each side of the polygon the size of quantum distance.

Just try and draw a Circle in The Gimp (or in Paint.NET).

Now zoom 600% in to it. You will see it is actually a polygon, composed of thousands of little pixels, the same is true for "circles" in reality, they are composed of thousands of little quantum pixels.

So, Circles do not exist, they are just a math abstraction. And that is why we end up needing unattainable numbers like Pi to deal with the area inside of a circle... because there is no such thing as the area inside of a circle (because there are no circles).

The area inside of anything in reality, that looks like a circle, can be calculated by dividing it in "quantum pixel areas", and the adding those areas to get the total area inside of the supposed circle. Now, that would require a lot of effort, that is the reason we believe in circles, because they are a very useful abstraction (even if it is an abstraction that fails and produces Pi as a result of its failure)

Among Archimedes mathematical accomplishments is the computation of pi, he used the method of exhaustion as a way to compute the area inside a circle by filling the circle with a polygon of a greater and greater number of sides. The quotient formed by the area of this polygon divided by the square of the circle radius can be made arbitrarily close to Pi as the number of polygon sides becomes large, proving that the area inside the circle of radius r is Pi * (r*r),Pi being defined as the ratio of the circumference to the diameter.

When a circle's diameter is 1 unit, its circumference is Pi units. Why Pi is such simple concept but such a strange "number"? because Circles do not exist, Pi is an example of a Mathematics (Geometry?) abstraction leak.

As we increase the number of sides, we approximate more, and more to the circle, but we can never really have a circle.

This is why we can never know the complete value of Pi, because there is now way to have a circle in a discrete quantum based reality, "circles" are a good model for this things with too many sides to count easily, but circles are just a model, not a reality

That is something, that as software developers we should always keep in mind: The Map (the Model) is not The Territory (the Reality)

Pagination in Dataphor

First I created this operator:

create operator GetNext(var AID :Integer) : System.Integer 
begin 
AID:= AID+1; 
result:=AID; 
end ;

and then I used it:

var id := 0; 
select SomeTable add {GetNext(id) rowid}; 

It is nice, but it would be better if there were a way to do this without having to create an operator myself (it should already include one that does this)

Now If I want to do pagination, all I have to do is write:

var id := 0; 
select SomeTable add {GetNext(id) rowid} where rowid between 2 and 5;

That will return rows between 2 and 5 from the “SomeTable” table.

Now one of the most interesting feaures about Dataphor, are relvars (relational variables), that make it possible to do this:

var SomeComplexQuery := SomeTable where …...
var id := 0;
select SomeComplexQuery add {GetNext(id) rowid} where rowid between 2 and 5;

That way, pagination can be added to a query of arbitrary complexity with minimum effort.

Dataphor: How to create a table with a primary key

Creating a table in Dataphor is easy:

create table Debt  
{  
    ID : Integer,

    key { ID },
};

That translates (when using MSSQL as storage) into (if we ask MSSQL to provide us with the SQL DDL):

CREATE TABLE [dbo].[Debt](   
    [ID] [int] NOT NULL 

)
 
CREATE UNIQUE CLUSTERED INDEX [UIDX_Debt_ID] ON [dbo].[Debt] 
( 
    [ID] ASC 

)

It is important to note that D4 does not have the concept of a primary key, and that the “key” keyword results in a unique index, that while for all uses and purposes can work as a primary key, it is not recognized as such by MSSQL. It is possible to have several different “keys” in a Dataphor table.

Scala is not mature enough for Weld

Scala is not mature enough for Weld (or any other reflection related operation). Turns out that one of the feature that make Scala so interesting (closures) is also a source of incompatibility (the internal anonymous class files that generate are invalid)

So, if for example, I have a code like this (inside any method in a class):

val classNames = Conversions.convertList(names).reduceLeft[String] { (acc, n) =>
       acc + ", " + n
}

That generates an inner class that makes Java reflection unusable:

java.lang.IncompatibleClassChangeError: com.googlecode.solder.dwr.WeldContainer and com.googlecode.solder.dwr.WeldContainer$$anonfun$getClasses$1 disagree on InnerClasses attribute
 at java.lang.Class.getDeclaringClass(Native Method)
 at java.lang.Class.getEnclosingClass(Class.java:1085)
 at java.lang.Class.getSimpleBinaryName(Class.java:1220)
 at java.lang.Class.getSimpleName(Class.java:1112)
 at org.jboss.weld.util.Names.typeToString(Names.java:251)
 at org.jboss.weld.util.Names.classToString(Names.java:263)
 at org.jboss.weld.introspector.jlr.WeldClassImpl.<init>(WeldClassImpl.java:151)
 at org.jboss.weld.introspector.jlr.WeldClassImpl.of(WeldClassImpl.java:133)
 at org.jboss.weld.resources.ClassTransformer$2.call(ClassTransformer.java:72)
 at org.jboss.weld.resources.ClassTransformer$2.call(ClassTransformer.java:68)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
 at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 at org.jboss.weld.util.collections.ConcurrentCache.putIfAbsent(ConcurrentCache.java:125)
 at org.jboss.weld.resources.ClassTransformer.loadClass(ClassTransformer.java:67)
 at org.jboss.weld.bootstrap.BeanDeployer.addClass(BeanDeployer.java:59)
 at org.jboss.weld.bootstrap.BeanDeployer.addClasses(BeanDeployer.java:86)
 at org.jboss.weld.bootstrap.BeanDeployment.deployBeans(BeanDeployment.java:134)
 at org.jboss.weld.bootstrap.WeldBootstrap.deployBeans(WeldBootstrap.java:367)
 at org.jboss.weld.environment.servlet.Listener.contextInitialized(Listener.java:158)
 at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3843)
 at org.apache.catalina.core.StandardContext.start(StandardContext.java:4342)
 at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
 at org.apache.catalina.core.StandardHost.start(StandardHost.java:719)
 at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
 at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
 at org.apache.catalina.core.StandardService.start(StandardService.java:516)
 at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
 at org.apache.catalina.startup.Catalina.start(Catalina.java:578)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
 at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)

Apparently this is a known problem since 2008-08-03 (16 months so far) and AFAIK they are not in a hurry to fix it.

I guess that means I am going to have to start removing Scala from my Weld project. I am sad about this because Scala is IMO a really beautiful language, that feels much cleaner than Java, but if the .class files that it generates are not compatible with the rest of Java... it is just useless for me.

Playing with Weld and Scala to create a new framework for JSPs

Today I built my first "serious" example using Weld and Scala.
You can take a look at it in here.
So far, Weld looks like really good framework, I am still not able to say that it will replace Spring as my "glue framework", but that could certainly become true.
Scala also looks interesting (this also my first "serious" attempt at doing something with Scala), but the support for Scala in Eclipse is very fragile... the plug-in still needs a lot of polishing (no refatoring, no quickfixes, and the syntax coloring and cursor control sometimes goes crazy). I would not recommend the Scala plugin in Eclipse for "real work", but for hobbing is fine. (I wonder if the support for Scala in the Netbeans or the opensourced version of IntelliJ is better).