Estimation: Is assuming the mother of all estimation errors? Or perhaps it is "to suppose"? Or maybe it is just our own ignorance?

I recently read a tweet that said:

Congratulations to "Assuming" for being the MOTHER OF ALL ESTIMATION ERRORS in software development!!!!

— SoftwareEvangelist (@vanessa_amaya) May 10, 2013

And I wondered: does the author realize... that she is assuming that assuming is the problem?

First, let's review the meaning of the word:

assume: assume. (From Lat. assumere).

1. tr. To take to oneself, to take for oneself.
2. tr. To take charge of something, to be responsible for it, to accept it.
3. tr. To acquire, to take on a greater form.

In software, the meaning we commonly use based on my experience is "To take charge, to be responsible for something, to accept it." What do we take responsibility for? The assumptions we use to build our estimate. Should we instead seek to be irresponsible? In my opinion, the problem is rather the assumptions themselves. If the assumptions do not align with reality, we end up delivering things poorly, for example in the scenario Ezamudio discussed in the previous post of this series, the problem escalated because the technician assumed he had chosen the correct part; in software development, it is more complicated, as there are several intertwined assumptions. But returning to the beginning, is assuming really the mother of all problems? No. Is supposing then the mother of all problems? I don't agree with that either, let's review the meaning of suppose:

suppose. (From Lat. supponere).

1. tr. To take for granted and existing something.
2. tr. To pretend, to give an ideal existence to what really does not have it.
3. tr. To entail, import. The new acquisition he has made implies excessive conservation expenses.
4. tr. To conjecture, to calculate something through the indications one has.
5. intr. To have representation or authority in a republic or in a community.

The meaning we commonly use in software, (at least those of us involved in its construction) is to conjecture, to calculate something through the indications one has, while we consider that the client sees it more as "To take for granted and existing something." Contradictory? Superficially, one might think so, but if examined more deeply, there is no contradiction. Supposing is a necessary something, we get out of bed, and we suppose we are awake, we drink water from the jar and suppose it is not toxic, we go out and drive to work and suppose it will be relatively safe. We do not have absolute certainty, but we also do not torment ourselves, we act taking it for granted, and if in the end the water or food causes us indigestion, we deal with it, if we start having indigestion several days in a row (if it does not kill us) we will change our diet and the water we drink. The same applies to software assumptions.

Supposing is a necessary activity in the scientific method (step 3):

1. Observation: Observing is applying the senses attentively to an object or a phenomenon, to study them as they really present themselves, which may be occasional or causal.
2. Induction: The action and effect of extracting, from certain observations or particular experiences, the particular principle of each one.
3. Hypothesis: Formulation through observation following the norms established by the scientific method. <---- ASSUMPTION!
4. Test the hypothesis by experimentation.
5. Demonstration or refutation (antithesis) of the hypothesis.
6. Thesis or scientific theory (conclusions).

Basically, we Observe, Analyze, Assume, and Test. If it works, we keep the assumption, if not, we repeat until we find the "correct" assumption. Now, this is the most important thing about the scientific method: We never reach the correct, the path to truth is like the limit in infinitesimal calculus, we are getting closer each time, but we never arrive.

The most important lesson of the scientific method is implicit: It's a Sand Mandala. Assumptions are made, only to be replaced by better ones. Thus, in software, assuming our assumptions are true is not a mistake, it's part of the method; the mistake comes when we forget that we are just at step 3, step 4 is still to come, where, without neglecting our responsibility, we test if our assumptions match reality. Therefore, assuming and supposing are not the mother of all problems, the mother is rather the lack of experimentation, and this lack of experimentation stems from treating assumptions incorrectly.

The first rule to follow for assumptions, again a rule from the scientific method, is that the assumption must be refutable (principle of falsifiability), what does this mean? There must be a concrete way, through a particular observation, to declare the assumption (provisionally) true or false. (An rrefutable assumption would be one that requires an exhaustive search of all possibilities to refute it). For example, I've been given RFPs to make systems that say "the system must implement all the relevant business rules for the business process to be automated" and then, nothing, no list of which rules are. Demonstrating whether we are meeting or not meeting that assumption cannot be refuted, the list of rules is potentially infinite (and clients certainly take advantage of that). What can we do?

When I was younger, what I used to do was demand the list of rules, or say that I couldn't do anything and refuse to estimate the time that would be necessary. Needless to say, that attitude only got me into trouble with my bosses, lost business, and dissatisfied clients. Although that's the path indicated in estimation books, it's not the correct path in the real world.

Of course, there are clients who are willing to give you the list of rules, so when you're in this situation, don't fail to ask for the list, but if they can't (or won't) give it to you, don't suffer, accept it, and continue... but cautiously. A few months ago, I watched a nascent consultancy suffer a horrible death and enter zombie mode: they sold a system with open-ended assumptions, according to them, it was going to be finished in 2 weeks; when I met them, they had been working for free for the government for a year, under threat of a lawsuit for breach of contract if they were to "crack" and not finish implementing all the rules that the client might think of. I couldn't help them anymore, what they needed was a good lawyer. How then to avoid falling into that hole?

Some would say: Simply reject the business. Yes, it sounds nice when you have a lot of money, have no responsibilities, and if you don't receive income you're the only one who doesn't eat. But what if your family depends on you? Can you really afford to reject the work? Of course not. Don't reject it then, nor tire yourself explaining why you can't "implement all the relevant business rules for the business process to be automated", the right answer is: define the scope yourself.

Place an assumption in your proposal in the following way: "It is assumed that these business rules will not exceed 10. It is assumed that the list of all business rules to be implemented will be received X days after the project has started. The rules must be defined as algorithmic step-by-step procedures, each consisting of no more than 10 CFP (Cosmic Function Points according to the standard ISO/IEC 14143/1:2003). It is assumed that all the information to be used by the business rules is that which is fed into the system through the user stories defined in the scope section of this proposal. If the list of business rules is not received at that time, or if a greater number of rules are received, the scope change procedure described in this document will be executed, which may have an impact on the project times and costs." There, you've defined the scope.

By doing this, essentially you are answering the question that some clients would surely ask you when you told them that you couldn't "implement all the relevant business rules for the business process to be automated". Some clients would ask: Why not? And you would tell them: Because I don't know how many there are, nor how complex they are, and you're only giving me 3 months to do them. Well, the client would say: How many can you do in 3 months?

And there is where, when I was younger, I would clash most strongly with reality: I simply couldn't answer that question! I could only say "because not", when the right answer is to say "as if". This is not a silver bullet, there will be clients who are not willing to accept a defined scope. In those cases, if the client is not willing to pay for time and materials, then yes, the right answer is thanks, but no thanks, I can't do the project.

In the next part of this series, we will delve into the benefits of assuming, and why the myth has been generated that one should not assume.

 Originally published in Javamexico

Estimation: Why do we do it?

 

There are those who consider that estimating is impossible and a waste of time, a useless endeavor. There are those who believe it can be done, but only under certain conditions, and there are those who think that the secret lies in following a certain method... However, before discussing these points of view, I want to focus on a question often omitted in articles and books on estimation:

Why do we estimate? And I'm not talking about the theoretical reasons typically used in books on the subject, but rather, out there, in the real world. I don't want to generalize, so what I will say next is strictly based on my own experience.

We don't estimate to know how long the project will take; the client usually has already set a deadline that is unlikely to change.

Why do we estimate then? We estimate to see if we can do something within the time and budget that are already established and that sounds to the client like what they asked for (because if one thing is certain, it's that most clients don't really know what they want until they've seen a couple of progress demos).

The agile community tells us: don't estimate, do everything by time and materials, although in reality what it's saying is "don't estimate beyond the next sprint" and "lose any client who is only willing to buy projects at a fixed cost and time." It would be nice if the world were that easy, but the fact is that it's not: most of the clients I've encountered in the consulting world want a budget upfront... Those of us who would like everyone to understand Scrum sometimes find it absurd, a display of their ignorance some say, but let's put ourselves in their shoes: who, when taking their car to the shop, expects to be told: we're going to fix your car iteratively, give us your bank card with an open voucher and then check your balance to see how much it was at the end...

And yet we go to the doctor, and while each visit may have a fixed cost, we don't ask the doctor to give us the cost of the medicines before diagnosing us, nor do we force them to tell us the maximum number of consultations we'll need... Medical treatment is completely iterative... Maybe one day we'll recognize the similarities between curing the most sophisticated biological system we know and creating or composing new systems... In the meantime, let's think... Why are there fixed-cost projects that do work? The level of uncertainty clearly visible to the educated mind at the start of the project should guarantee that this never happens... What's different in those projects? Maybe it's not obvious, but in my experience, it's quite simple: the scope was better defined, either because the client themselves had already done some pre-analysis, or because the consulting team did the same... SACRILEGE! Am I daring to say that the correct way to carry out a project is with Waterfall?

NO! To believe that is to ignore the principle of diminishing returns: those who believe that this indicates Waterfall is the answer are making the mistake of thinking that if some analysis is better than none, then double or quadruple the analysis will bring a benefit directly proportional. Well, welcome to the real world: things don't work like that, if you do a lot of analysis, you also consume a lot of time and while your understanding of the situation becomes more precise, it also becomes progressively outdated... The world doesn't stop changing because you analyze a particular point of it in time and by the time you turn to compare your analysis with the current reality, the world has already changed: the conclusions of your analysis are now obsolete.

Zero analysis? Bad. Exhaustive analysis? Bad... What to do then? Find out how much is "enough" but first understand enough for what? Many books on estimation talk about achieving 75% accuracy, as the ultimate goal of a well-done estimate. In my experience, that perception is wrong. What good does it tell a client: to succeed in what you're asking, I need one million pesos, if the client only has (or claims to have) a third of that? It's no use, the client won't have their product, and we won't have our project. What to do then?

Look for the "as if," the first time I heard that phrase it seemed naive: do a one million project for a third? Impossible! But then it happened that some people asked me: why? And I complicated my life giving a thousand explanations about the complexities that are part of software development (many of which I didn't understand then... and many of which I still don't fully understand now) and... I couldn't convince them, and ended up trapped in horrendous Dead March projects from which I not only received pressure from the client, desperate because the project was nowhere near completion by the deadline, but also from the company I worked for, equally desperate as the project's margin eroded away...

And after finishing that project, we embarked on another just like it... How do we break that cycle? Is  it true, as I recently read on Twitter, that assuming is the mother of all problems in system development. I will talk about that in my next post...

 Originally published in Javamexico

SQL 2012 Bug: NVarchar changes where evaluation order

Today I arrived late at my house because my team had a big problem migrating a stored procedure from SQL Server 2000 to SQL Server 2012. Countless hours lost trying to find out what we could be have been doing wrong...

On the end, in turned out, all the trouble was because of a bug in SQL Server 2012.

Here is what we found, lets say you have a table "T_1":

CREATE TABLE [dbo].[T_1](
[C] [nvarchar](50) NULL
)

Now, lets say you add some rows to it.

INSERT INTO [dbo].[T_1] ([C]) VALUES ('P')
INSERT INTO [dbo].[T_1] ([C]) VALUES ('Q')
INSERT INTO [dbo].[T_1] ([C]) VALUES ('R')


Now write this query:

select * from T_1 where ISNUMERIC(C)=1 and CONVERT(float,C)=0.0

And you will get a nice error message:


Msg 8114, Level 16, State 5, Line 1
Error converting data type nvarchar to float.

¿What is the bug? You shouldn't be getting an error message! the "Error converting data type nvarchar to float." is generated by the CONVERT(float,C)=0, but SQL should never run that code because ISNUMERIC(C)=1 evaluates to false, and if the first part of an "and" is false, there is no point in executing the second part, the result is going to be false anyway.


Well you might say, maybe SQLServer has always done things this way... well, no, it has not. In SQL 2000, that query executes correctly!

And in SQL 2012, there is a workaround, just change from nvarchar to varchar:


CREATE TABLE [dbo].[T_2](
[C] [varchar](50) NULL
)


INSERT INTO [dbo].[T_2] ([C]) VALUES ('P')
INSERT INTO [dbo].[T_2] ([C]) VALUES ('Q')
INSERT INTO [dbo].[T_2] ([C]) VALUES ('R')



Now if we write (note we are now working with the table T_2 that uses varchar) :

select * from T_1 where ISNUMERIC(C)=1 and CONVERT(float,C)=0.0

We will get no error.

Why is this happening then?

It seems to be a bug in SQL Server 2012 execution plan:

 

 

As you can see in the image, SQL 2012 inverts the predicate when working with nvarchar!

On the other hand, for T_2  the table with a varchar column:

As you can see, here the the order of evaluation is preserved, and things work like they should.

Now what can we do if we are not allowed to change the type of the column in the table?

I tried using a CTE, but it does not work, the execution plan is the same faulty one:

with V_1 as (select * from T_1 where ISNUMERIC(C)=1)
select * from V_1 where CONVERT(float,C)=0.0

Using a subquery also fails:

select * from (select C  from T_1 where ISNUMERIC(C)=1) V_1 where CONVERT(float,C)=0.0

We can change the type explicitly to varchar in the query, that fixes the problem:

select * from T_1 where ISNUMERIC(C)=1 and CONVERT(float,convert(varchar(100),C))=0.0

But,  what if we actually have an Unicode string with chars that will get damaged by a conversion to varchar?

I shouldn’t be a problem…. What do you think? any other workaround?

UPDATE: I have submitted this bug to Microsoft Connect, click here to see my bug report

Web Slices: you need to use Alternative Display Source or javascript will not work

If your WebSlices use the Basic Web Slice model, javascript will not work (this file is WebSlice.html):

<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server">
    <title></title>
</head>
<body>

<div class="hslice" id="SliceID">
     <span class="entry-title">Title of the web slice</span>
     <div class="entry-content">Preview of the <a href="#" onclick="document.getElementById('Message').innerHTML='Hello'; return false;">web</a> slice
       <div id="Message"></div>
     </div>
     <p>
     Hola Rebanadas Web!
     </p>
  </div>

</body>
</html>

you have to use Alternative Display Source to make javascript work (this file is WebSlice.html):

<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server">
    <title></title>
</head>
<body>

<div class="hslice" id="SliceID">
      <span class="entry-title">Title of the web slice</span>
      <a rel="entry-content" href="AlternativeDisplay.html" style="display:none;">Alternative Display Source</a>
      <p>
      Hola Rebanadas Web!
      </p>
   </div></body>
</html>

 

and in the file AlternativeDisplay.html you put the code that use to be in WebSlice.html (and that needs to use javascript):

<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server">
    <title></title>
</head>
<body>
     <div class="entry-content">Preview of the <a href="#" onclick="document.getElementById('Message').innerHTML='Hello'; return false;">web</a> slice
        <div id="Message"></div>
      </div>

</body>
</html>

How to migrate your local user profile to the domain

Apparently, there is no easy way (this or this do NOT work), there use to be a tool to do this (Moveuser.exe), but it stopped working with Windows Vista.

The way to do it now is to write a VBScript that uses WMI, happily, I have found someone that has already done it here.

It basically seems to be using the ChangeOwner Method of the Win32_UserProfile Class, one problem I have found so far is that if the profile to be “moved” in to the domain is heavy (many Gbytes of weight) the ChangeOwner method can take a really long time to do its job. A possible solution then is to move the files outside of the profile dir, run the migration process, and copy the files back (you might need to change the permission information of those files to be able to copy them back)

How to move C:\Users to D:\Users

There is no user friendly way to move your C:\Users folder to D:\ but the user unfriendly way to do it is not too hard (it was a little hard to discover, I learned how to do this when I decided that I wanted to have C:\Winnt\Profiles dir at a different location, after I lost most of my data when my main disk failed, back when Windows NT 4.0 was the latest and greatest OS from Microsoft)

Basically, after you installed Windows (this instructions are for Windows 7 but the general approach is valid since Windows NT 4.0)  you have to:

1. Login with “WhatEverIsYourUserName” account.
2. Run the Command Prompt (cmd.exe) as an Administrator
3. 
4. Copy the C:\Users\ folder to D:\Users using Robocopy
   

   robocopy C:\Users D:\Users /E /COPYALL /R:0 /Z /XJ

   
   



5. Open regedit and modify the ProfileList entry so that it looks like this:

1. Create a new (Administrator) Windows account that you will use to test the configuration change
   
   
    



2. Restart



3. Login with the “Test” Windows account



4. Delete C:\Users



5. See that the  Start menu still works (if you made a mistake, your Start menu is now empty)



6. Delete the registry Key for your “WhatEverIsYourUserName” account (if you still want to use it). (The registry Key is the yellow folder with the S-1-5… name that contains the value ProfileImagePath that points your now obsolete C:\ folder, remember,  you have to delete the whole folder/key, not just the ProfileImagePath value):
   
   
    



7. Log out



8. Now log in with your “WhatEverIsYourUserName” account



9. If you take a look at the registry now, it will look like this (The value in ProfileImagePath now starts with “D:\” instead of “C:\”):
   
    



10. Now you can delete the test account.

And that is it, now the users accounts live in D:\Users\

Recover sa account

Today, a friend from the office locked himself out of SqlServer 2008 R2 Express, somehow he managed to remove the windows Administrator account form the list of sysadmin accounts in the SqlServer express installed in a development server.

I thought there was no way to recover from a mistake a like that, but another friend found the way and gave us a link with the solution.

Basically, we need to stop the SqlService, and restart it in "single user mode" and "minimal config mode"

sqlservr.exe -m -r -s SQLEXPRESS

And the open another command window and run this commands:

osql -E -S .\SQLEXPRESS 
exec sp_password @new='changeme', @loginame='sa' 
go 
alter login sa enable 
go 
exit 

And that was it. Remember, without the special "-m -r" options, the osql commands will fail.

NOTE: In case your SQL Server is not configured to use Mixed Mode for authentication, you may also need to make some modification to the registry settings.

Basically you need to find the registry entry for the instance:

Default instance:

HKLM\Software\Microsoft\MSSqlserver\MSSqlServer\LoginMode

Named instance:

HKLM\Software\Microsoft\Microsoft SQL Server\Instance Name\MSSQLServer\LoginMode

And  change the value of LoginMode to 2.