JPA Myth: EclipseLink and Hibernate are compatible. Not true: Hibernate does NOT follow the JPA Spec

Please take a look at the Hibernate JIRA issue EJB-441, seen it?

Now, please tell me if this is correct:

According to Gavin King (author of Hibernate) the JPA spec the @Column(nullable=false) is just a schema generation hint, and therefore JPA implementations should NOT use it for object level validation.

I think that means that if an entity object with a null property marked with the @Column(nullable=false) is persisted using entityManager.persist() it should crash with a java.sql.SQLException as the root exception, thrown when the "INSERT" statement failed after it was sent to the database by the JDBC driver (as it happens in EclipseLink), and not because some implementation specific mechanism "pre-validated" it and prevented it from reaching the database (as it happens in Hibernate thanks to org.hibernate.engine.Nullability.checkNullability).

Is that correct? Because I don’t know anymore!

Lets say that is correct, does that means that the following is correct too?:

Validation should be entirely the responsibility of the JSR-303, and the properties of an entity should only be checked for nullability before sending to the database if a JSR-303 implementation is there to indicate that.

Is that true? And if it is, then:

What is exactly the difference between @Basic(optional=true) and @Column(nullable=true)? What happens when this 2 are applied to the same property, but they contradict each other? (As in : @Basic(optional=false) and @Column(nullable=true) or in @Basic(optional=true) and @Column(nullable=false)

I could not find a statisfactory answer in the spec documentation, would anyone be so kind as to resolve this issue? Because it Hibernate Forums they have decided that the way to deal with this inconsistencies and contradictions in Hibernate and in the JPA standard is to plain ignore me since May of 2009.

As you can see I tried to raise the issue with 2 JIRAS, forum post and easy to reproduce unit-tests, but nothing seems to work… what should I do then?

EntityManager.persist: ¿What does/should it mean?

Lets say you are presented with the following JPA @Entity:

@Entity
public class Customer {   
  private Long id;
private String name;
  @Id
@GeneratedValue(strategy=GenerationType.AUTO)
public Long getId() {
    return id;
}   
  public void setId(Long id) {
    this.id = id;
}
  @Column(nullable=false)
public String getName() {
    return name;
}   
  public void setName(String name) {
    this.name = name;
}
}
 

And then the following test

   @Test
public void persistCustomerInTransaction() throws Exception {
    factory = Persistence.createEntityManagerFactory(PERSISTENCE_UNIT_NAME);
      EntityManager em = factory.createEntityManager();
      em.getTransaction().begin();
      Customer customer = new Customer();
      em.persist(customer);em.getTransaction().commit();
      em.close();   
  }

And the following question: Where does it crash?:

1. At the line em.persist(customer); because the name is configured as nullable=false, and we are trying to persist the Customer instance with a null value in the name property



2. em.getTransaction().commit(); because the name is configured as nullable=false, and we are trying to commit the transaction with a null value in the name property

Turns out… that the answer depends on the JPA provider you are using!!!! If using Hibernate, it crashes at em.persist, but if using EclipseLink, it crashes at em.getTransaction().commit.

Now, this might seem irrelevant, but it is in fact very important, because EclipseLink behavior means that constraint validations are deferred until the point where the transaction is committed, and that means that the developer has a lot more freedom to manipulate its persistent objects: they can be in any (possibly invalid) state while they are being manipulated by the business logic of the application, and they only have to be “right” the moment one needs to commit them to the database (not before), this is specially useful, for example, when building a wizard like UI (or a plain simple CRUD UI with support for full object graphs) with EclipseLink, I can persist my objects as soon as I want, and if want to, for example, run some validation logic at the end, all I have to do is ask the EclipseLink to give a list of all the objects that are going to be written in to the database, with Hibernate, the entityManager does not help me manage new object instances, I have to manage them myself.

What I find really surprising is that AFAIK there should be some kind of test to ensure that all JPA are compatible (something called the TCK?), and I think that this kind of discrepancy should be detected by those tests... shouldn't it?


I think this support for deferred validation, and an API to get will be inserted, will be updated or will be deleted, will be very important to really integrate JPA with something like JSR 330, so that validation can really integrate with the lifecycle of a persistent POJO.

JPA/Hibernate (subjective?) weaknesses

I have had to work with JPA/Hibernate for a few years now... and I feel it has some weaknesses I really do not like (when compared with the first ORM I ever used, NextStep EOF), I am thinking about "switching to something else" but first i would like to be sure that the ORM I switch to does not have this weaknesses too.
List of (subjective? perhaps I only perceive them because I was exposed to EOF first?) weaknesses in JPA/Hibernate:

• No way to manage an object that "will be persisted", in JPA/Hibernate if you call entityManager.persist(object) and the database does not support sequences (like MS-Sql) an insert will be triggered, and if any of the non nullable fields of the objects is null, it will crash. If your object has a compound primary key, things get worse, because the entityManager can not deal with it until the compound primary key is set, and if your compound primary key is formed by foreign keys pointing to objects that are new too, that means you will not be able to save stuff with with a simple single call to entityManager.persist to one of the objects, cascading will not help you (I really miss something "magic" like the single shot EditingContext.saveAllChanges() in EOF)
• No easy way to know if "an object is dirty" (if it has changed/deleted since it was read from the database, there is just no API for that), and since you can not know what objects will be persisted, and what object have changed, and what objects will be deleted from database, that means you can not easily create an unified API for centralized polymorphic validation (that is no easy way to create validateForSave, or validateForDelete methods in your persistent entity classes)
• No real equivalent for validateForXXX, JPA lifecycle callbacks are not match for validateForXXX because you can not query the database during the lifecycle callbacks, and if you throw an exception inside a lifecycle callback, the JPA/Hibernate entityManager enters an invalid state, and after that you can not continue to use your POJOs, you have to start over with a fresh entityManager... and without you modifications to the POJOs. Note that Hibernate new validation framework does not offer a real solution for this problem... and AFAIK JSR-000303 will not help with this either.
• No support for some kind of temporary id: In JPA/Hibernate, the id for an object is "null" until you flush it to the database, so if you need to reference a particular object instance from the user interface... there is plain no way to do it, you have to "save it first" to get a primary key.
• No support for Nested Contexts... ( I think they would be a perfect fit for conversational frameworks like Seam or Shale). One of the goals of the ObjectContext is to provide an isolated area where local object changes can be performed without affecting other similar areas or the underlying storage. Nested Context changes can be saved to the parent Context without saving them to the database. Such child context is often called "nested". Nested contexts are useful in many situations, such as nested UI dialogs, complicated workflows, etc.
  

Those are my main disagreements with the way I have to work with JPA/Hibernate... will switching to JPA/EclipseLink, JPA/OpenJPA or the still not fully JPA compliant Apache Cayenne help me with those? I will be writing about my findings on this in my following posts

Ruby is younger therefore better than Java?

I wrote this as a response to From Java to Ruby: Programmer's view, but I couldn't post it because of a bug in that site, so I decided to post it in my own blog:

Isn't this a simplified view of the advantages of Ruby over Java? For example the lack of choice in Ruby means that if the "one Ruby way" to do stuff is not good for you project... you will have to go to other technologies (Java for example). And that can happen pretty often:

• Hibernate has many more options for integration with legacy databases than Ruby's Active Record... almost all databases now have JDBC drivers (you can not say the same about Ruby database support).
• Spring offers integrated transaction handling that makes it possible to switch from JDBC transactions to JTA transactions without changing a single line of Java code (you just need to modify around 5 lines in an XML file) what is the equivalent for that in Ruby?
• Calling stored procedures is not that easy with Ruby... what is the equivalent of HQL (JPAQL) for Ruby... can you honestly say that it can handle all the special cases HQL can... and with the same efficiency?
• With java I can build a web application GWT style, JSF style or plain JSP style... and each style has advantages and disadvantages... do I have all that power with Rails? (Of course, those frameworks can be re-built in Ruby, but the question is, do I have them now?)

I think you are right when you say that it's healthy to start with a clean slate and rebuild based on a cleaner, simpler foundation, but before saying that the new foundation is actually better than the older one, you have to be sure that your new foundation is actually capable of handling all the special cases the old foundation was capable of handling... or remember that maybe if you remove all the abilities the old foundation has to handle special cases you might end up realizing that your new foundation is just a replica of the state that the old foundation had when it was younger. (And even then the older foundation has the advantage that you can use it as it was used in the past, but you can not use a new foundation as it will be used in the future)

Unit testing Relational Queries (SQL or HQL or JQL or LINQ...)

Hi!
Most applications I have built have something to do with a database (I remember that while I was on college I used to think that was not exiting stuff, I used to dream about doing neural networks stuff, logic programming in Prolog, etc) but then I met WebObjects and its Object Relational Mapper (Enterprise Objects Framework) and I got really excited about object oriented data programming... but I always had a problem... to test my object oriented queries I had to "manually" translate them into SQL and test them against the database, and only after they give me what I thought were correct result I would write them using EOF Qualifiers...
Then I met Hibernate's HQL and I realized it is much more powerful than EOF Qualifiers, but I still had to translate it to SQL to test it, I know I can get the SQL that is generated from the HQL from the debug console, and paste it in my favorite SQL editor... but even then if I found a mistake, a lot of times it was easier to tweak it in SQL and the manually translate it HQL.
Currently, there are some extensions for Eclipse (Hibernate Tools) that make this more "direct" but, what if I don't like (or don't want, or can't) use Eclipse... it would be great if someone could, for example, make a plugin for SquirrelSQL, but until then... what options do I have?

Then I learned about unit testing... and the answer came to my mind immediately: I just had to write a unit test for each of my queries. That worked fine... in the beginning... until I started having queries that returned thousands (or millions) of objects, and it wasn't such a good idea to output them to the debug console... and I had another problem... how should I write the "asserts" of query?... and how can I do it so that it doesn't make my test so slow that it becomes unusable? (I can, of course, check the results just by viewing them, but my brain is not that good to say if those 10,000 row really match with the idea I had when I wrote that HQL)

So, I started to look "what do I do" to check if an SQL query is correct, lets say for example, that I write this:

select count(*) from Address,Employee where Address.Id= Employee.AddressId and Employee.Id = 3

(Translated to English: How many address the Employee with Id = 3 has?)

Now... how do I test that? well I could add an assert after getting the result in java (or c#) like this:

assert(count>0)

But the, what happens if someone deletes the row with Id = 3 from the table Employee? That means my test will fail... or what what if someone deletes all the addresses from employee? and what if I want to test that if there are no addresses for an employee, the answer should be zero...

That is a lot of work just to test if that simple query is right... and I think that work could be done automatically:

Take a look at the SQL, it could be decomposed into:

select count(*) from (select * from Address,Employee where Address.Id= Employee.AddressId and Employee.Id = 3) as EmployeeAddresses

And then we could say, lets automatically check for the case when the resulting set is empty, and for the case when the result is not empty, to check for a case when the result is empty, we need and Employee with Addresses, so we generate:

Select * from Employee where exists(select * from Address where Address.EmployeeId = Employee.Id)

And take the Id first employee we get... and that should give us a non empty set if used in the original sql sentence that we are trying to test... after that, we automatically generate:

Select * from Employee where not exists(select * from Address where Address.EmployeeId = Employee.Id)

And take the Id first employee we get... and that should give us an empty set if used in the original sql sentence that we are trying to test...

I call this queries "inverses" of the original one, it like when one is testing a multiplication, to see if 2 x 3 = 6, just do: 6/3 = 2 and 6/2 = 3, if 2, and 3 match the operands of the multiplication, you multiplication is right. The same thing goes for SQL, one just has to find the way to "invert" it, if I could automate this inversion, the automatically generated queries would help me by telling me things that might not be immediately obvious to me when I look at the original query, and that would help me check if my original query is right.... it would me some kind of "invariants" that would help me to better understand my querying... or maybe I could even write the invariants first, and then create a query and see if it matches my invariants...

Mmmm.... maybe using a select there is another way to "invert" a query to test if it is right, using the actual inverse operation of selecting... that is "inserting", could I derive from:

select count(*) from Address,Employee where Address.Id= Employee.AddressId and Employee.Id = 3

Something like (In pseudocode):

Insert Employee;

Store Employee.Id

Run select count(*) from Address,Employee where Address.Id= Employee.AddressId and Employee.Id = EmployeeId

Assert("The answer should be zero")

Insert Address related to Employee

Run select count(*) from Address,Employee where Address.Id= Employee.AddressId and Employee.Id = EmployeeId

Assert("The answer should be one")

This has the advantage that I don't need a database with data already on it, but it has the disadvantage that takes lot of time to write an unit test like this in java, because to insert an employee, it might be necessary to:

• Avoid breaking validation rules no related to this particular test, for example, an Employee must be related to a Department, but if the Department table is empty, then I should create a Department or I will not be able to insert an employee.
  
• Avoid conflicts with validation rules directly related to this particular test, for example, what if I have an Hibernate interceptor that won't let me insert an address without 1 or more Addresses

The main problem here, I believe, is that f I insert a row leaving a not null column empty most databases won't wait until I try to commit the transaction to say "integrity violation" and rollback my changes... therefore it is impossible to write partial data just for the test that I have in front of me, but... could I automatically generate consistent inserts using as a source just the integrity rules at the database level... and the select that I want to test?
I think it can be done... the question is..
What is the algorithm to generate the inserts needed to satisfy an SQL select statement?

UI Design: Edit then List vs List then Edit

Today I was thinking about the way I build UIs (List then Edit) and the way most people I know build UIs (Edit then List) and the relation that the way you build you UI has with the way you store the data you manipulate in you UI.

List then Edit

In this "pattern" for UI structure, after the user selects an option in the menu he sees a window that allows searching with some controls (generally in the top of the window),and with some control in the bottom that represents the search results, then the user selects one of the search results (an object, or row if you prefer) and he proceeds to work (edit, modify) it, for that he opens a new window "the editor" (perhaps by double clicking the selected object, perhaps by clicking an "edit" button in the list window) in the editor he may modify the object being edited as much as he likes, and then click "save" if he wants to commit his changes or "cancel" if he wants to rollback them.
If the user wants to create a new object, the list has a "new" button that also calls the "editor" so that the user can set the initial values for the fields before the object is committed to the database for the first time

Edit then List

In this other "pattern" for UI structure, after we select an option in the menu we are right there in the editor, but all the controls that could allow us to modify the current object (or row) are disabled, because we haven't searched for any (or created a new one) from here the user can choose to create a new object that action has the effect of enabling all the control in the editor, or search for an already persisted object (or row) by clicking the "search button", that shows the search "list" window, in the top of that window there are controls to configure the search criteria, and in the top a control to represent the search results, from here the user can choose one of the search results (an object or row), and click on the "accept" button, which takes him back to the editor that now is displaying the previously selected object, here the use may choose to make some changes and click "save", or just cancel click "cancel" but actions have the effect of disabling the controls in the editor UI until the buttons "search" or "new" are used again.


Edit in List
When an application generally follows List the Edit, sometimes, if the info in the selected object is very simple, the user may modify it right there, but for for objects with medium to complex (several fields, to one or to many relations with other objects) calling a window to modify the object is more comfortable for the user

Search (List) in Editor
Some times, the developer likes to use the same UI to Edit and to Search, this works like some kind of "query by example", that is when user enters the editor, instead of being disabled an waiting for a click on the "new" or "search" button, the controls are enabled, and if the user writes some data in to them and clicks "search" the already persisted object (or row) that is more similar to the partial information already written in the editor is loaded. Sometimes there are many objects that match the query by example, here is when a navigator control can be a nice thing to have.

Master List, Detail Editor
This is also a very common configuration, both the list and the editor are in the same window, the list generally in the top, and the editor in the bottom, I think this somewhat corresponds with Two Panel Selector.

Persistence
Okey, until now I have exposed what I think are the some of the common "patterns" used to create UIs that manipulate data, now... what I find more interesting about this is the the way this "patterns" affect the way the data is stored or retrieved from persistence:

• If we use Edit then List:
  • The Edit is displayed on the screen
  • The user clicks "search" and the search List window appears
  • The user configures the search criteria in the List window
  • The objects (or row) matching the criteria are shown in the List window
  • The user selects one of the matching objects and click in the "accept button" in the List window
  • We return to the Edit window, that now is displaying the selected object.
  • We can make changes to the current object by clicking "save" or click "cancel" to dismiss the changes, if we do either action the Edit controls are disabled and we again have to click "search" or "new" to work with an object
  • The problem here, is that the user might want to see the same search results he used the last time, but each time we call the search List window, we are creating a new object, and all of the configuration from the last time we called it is already lost (this disadvantage has a "nice side", because we don't have to worry about showing "stale data" to our user)
  • Another disadvantage is that maybe not all the controls we use to manipulate the data in the edit windows are "databindable" so we have to manually reset the state of the Edit window, and if we don't do it correctly we could have bugs that "transfer information between edits". (First I edit John and set his age to 24, then I edit Mary, and her age is also shown as 24, but her age is 56, and we had no intention of changing it, it happened because we forgot to clear the text-box value)
    
• If we use List and then Edit:
  • The list is displayed on the screen
  • The user configures the search criteria in the List window
  • The objects (or row) matching the criteria are shown in the List window
  • The user selects one of the matching objects and click in the "edit button" in the List window.
  • The "Edit" window appears on top of the List window, and is displaying the selected object.
  • We can make changes to the current object by clicking "save" or click "cancel" to dismiss the changes, if we do either action the Edit window is closed and we return to the List window
  • One of the advantages of this approach is that we don't have to worry about the "transfer between edits" bug, because each time we call the Edit window, it is a new window, without any data, ready to configure itself to match the data contained in the object we are going to edit.
  • Then problem of keeping the search results to reuse them is also automatically solved, because the search List window was never closed, it was there all time, behind the edit window.
  • But the problem now is that some of data shown there may not be updated, or perhaps, it has data that has never been on the database, and that we do not want on the database, how can that be? well, we edited the object in the edit window, and the databinding ensured that the changes were written to the object before we clicked "save" or "cancel", if our intention is to keep those changes and we clicked "save" in the Edit window then we don't have a problem... but if we clicked cancel, now we need to rollback in memory changes and some of this changes could have modified relationships with other objects, or properties of the other objects, we could have created, deleted or updated a complex graph of objects but "only in memory" and now, we need to rollback all this changes, if we are using an Object Relational Mapper (ORM) that supports this (like Apple's EOF) or a relational cache that allows for in memory transactions (like the .NET DataSet) we can solve this problem easily (of course the DataSet has other disadvantages), but if we are using an ORM like NHibernate that AFAIK does not rollback in memory changes then you have a problem, you have to re-fetch the information from the database.

This seems like a big omission from the NHibernate guys... but it isn't exactly so, everything I have exposed here, has been on the assumption that we are working in a "Smart-Client" that holds local information, and Hibernate was born in "the web world". In the web, you "need" to re-fetch the information on each request (so when you go from editor to list or from list to editor you always reload the data) and you don't really pass the object you are going to edit from the list to the edit, it easier, and more efficient to just pass the primary key, of course the problem there is that you can only do that with objects previously persisted in the database, if you object is new it has to be serializable and some times that just isn't advisable (but that is an issue for another discussion)...

The thing with web based application is that the controls in the UI don't actually store the information directly in your object, their store it in the view-state, or in the query-string, in cookies, in the session or in an object that "represents the form", and only when you finally want to save, you extract the information from the web-controls and write it in to your object (at least that is more/less was the way I did it in the Java servlet world) but this is a "feature" that might be going away... the problem is that with the new JSF databinding, or with the new databinding facilities of ASP.NET 2.0, you can actually bind you controls directly with you datasources... and then how will we rollback the in memory changes? should we build a framework on top of NHibernate, a kind of "in memory object context" that handles the commits and rollbacks in memory?

More Questions

• And what about the case when a list window calls an edit window that has an embedded list control that calls another edit window? (complex multiple level or nested interfaces) should the object relational mapper make it easier for me to build this kind of UIs?
• Or Should a new kind of framework deal with the problem of communicating the persistent objects with the UI?
• is using DTOs really the solution? Does Apple's EOF go beyond the responsibility of an ORM by providing in memory transactions?
• If NHibernate can almost transparently persist objects to the UI, shouldn't this other framework be able do the same and transparently present the object in to the UI without having to manually create objects to do this job?
  
  I am thinking about publishing this post as an article in the Wikipedia to see how it evolves... I would love to read some comments about this article to improve it.