Person A: “He says that he wants a blog.”
Person B: “A Blog!? Have blogs been approved?”
Person A: “Wikis have been approved. We can give him a wiki.”
Person B: “Are you sure? You said that he wanted a blog.”
Person A: “He SAYS that he wants a blog. Let’s give him a wiki and tell him it’s a blog.”

Some people just don’t give a FLOSS. Lord love a duck!

I have read all sorts of crap that makes claims about the superiority of one database product over another. In my experience, I have seen some of the worst databases developed using the some of the best tools. Modeling is important, and so many of the databases I see are poorly conceived.

Lately, I have been thinking about data – particularly something I have been calling smart data, but I digress. The so-called relational database is based on set theory, which is a precise mathematical way of thinking about things. I say so-called not because I want to criticize the relational database or the theory that supports it. I want to criticize the way the tool has been used.

One of the things that strikes me is that it is hard to build a new relationship in most databases. You may have to build an association table or add a new field, but you also have to write SQL statements, stored procedures, and code to use the new relationship. There should be an easy way to say that a relationship AB exists between entity A and entity B; and, it should be easy to begin using this information without writing new code.

If I were to come up with a new way to define and use data, this would be a requirement. The relational database can store and define relationships, but it requires too much thought and too much work. In the world we live in, things are related to each other in a multiplicity of ways. There are all sorts of relationships – for example, some of the possible relationships between a book (entity A) and a person (entity B) are:

1. Was read by
2. Is being read by
3. Is owned by
4. Was borrowed by
5. Was returned by
6. Was lost by
7. Is loved by
8. Was given to
9. Is recommended by
10. Is being used by
11. Was sold to
12. Was stolen from
13. Was stolen by
14. Was found by
15. Was burned by

The relational database, as it is being used, makes it hard or impossible to create relationships on the fly. The work of creating new association tables, or adding fields, to express these relationships is prohibitive. Even if one were to build a database with a relationship table that allowed for multiple relationships between books and people, it would still be necessary to build a new relationship table to store the multiple relationships that could exist between the other entities in the database.

Basically, if I were to go out on a limb, I would say that the relational database is not relational. Each entity table in a relational database represents a class of entities, defined by a set of attributes, some combination of which uniquely defines the entity. If I have twenty entity types in my database, and I add a twenty-first, this could create twenty new relationship associations – my latest entity type could be related to any or all of the other twenty entities.

In practice, we usually only accommodate the relationships that are useful to us, and we live with the lie that the other entity types are not or could not be related to each other. It is too much work and coding to represent the data as it is. Data modelers are good at creating a physical model that can store and retrieve data is ways that are useful within the confines of some narrowly defined requirements.

For example, a database may provide the means to store the fact that employee X works at warehouse Y, but not the fact that warehouse Y is wheelchair accessible. To store this information, we can add a boolean attribute to the warehouse table to identify it as wheelchair accessible or not. Or, we can create an association of wheelchair accessible types with warehouses – this has the advantage of allowing us to associate wheelchair accessibility with any other facility that is not a warehouse. However, this all requires work by a DBA and some developers. Basically, as it is, the model is too rigid and brittle to accommodate changing requirements.

Relationships are important, but the relational database seems to focus more on attributes. Object databases, as they are being build today – to the best of my knowledge, that is – mirror the thinking that goes into building relational databases. Object stores are often used as a substitute for a relational database, and they are often compared to relational databases. I think this will change as people see the new possibilities that open up in an object-based repository.

Moving to an object store should mean moving to new ways of thinking about data. It should be easy to define a new relationship, to specify that it applies to the domain of entities of type A and type B, and to begin using the relationship without writing new code.

Using db4o, I have created an object that serves the purpose of storing information about the relationships that exist between entities. Any entity can be related to any other entity, but some relationships are only possible between certain types of entities. For example, I may allow books to be owned by people, but I may not allow people to be owned by people. I’ll see how well I can get this construct to perform – first, it has to be correct, but later, I want to it be fast. I have provide the means to associate any entity with any other entity in a multiplicity of ways.

Please note: E.F. Codd was smarter than I am – at the very least, he knew more about set theory. The relational model is not dead – nor should it be killed, but thinking about data is fun. Mistakes and errors are instructional. Correct me when I seem to be wrong.

On my last project, I was working on an application that had no architecture at all. To accomplish any thing at all, I was often required to play with SQL statements, connection strings, HTML, CSS, JavaScript and VB.NET. The business rules were implemented in each and every form of the application, and it was impossible to know what the impact of changing a single rule would be. It was hard to make progress in such a tangled mess of code.

I made the case for designing and applying a consistent architecture, but my client was more concerned about new features that somehow always had to be available right now, or with making consmetic fixes. We did a good job patching up the leaky ship that was their application – in fact, we undermined the case we were trying to make for stopping all new development in order to redesign and rebuild the old application. The current application seemed good enough for all intents and purposes.

I have been working on a new project for a few weeks, and I have been able to accomplish more for my client in a short time than I would have been able to in a couple of months for the last client. Although we are working with a database, I was not required to learn about its structure. I simply use the business objects that talk to the database. I can concentrate on the straight forward tasks at hand. Many of the objects I use provide functionality I do not even need to think about or know about to be productive.

What is more, the architecture is guided by a set of values and assumptions that are easy enough to deduce. I am having a good time.

I am currently working on an application that was very badly coded by another developer. Touching any part of the application is almost certain to embroil me in business rules, SQL statements, xhtml, Javascript, server side code or style sheets. Whatever I do, I almost always have to rely on my knowledge of the database to make a simple change to the application or fix a bug – there is no way around it. The database and the application are tightly coupled, and not even my client thinks of them as two separate entities.

I am inclined to make the decision that the business logic should live in the database, and that the database should be exposed to application developers by means of efficient and correct stored procedures. I am inclined to separate the responsibilities of team members:

• Web designers should build standards compliant web pages.
• Application developers should write server side code to fetch data from the database to be passed to templates that we built by web designers, or they should pass data to the database from the UI, responding to any errors that occur.
• A DBA should be responsible for building a database that performs well, is flexible and adheres to, expresses and applies business rules. And, the DBA should expose procedures that make the data available to developers

As far as I am concerned, the singers should sing and the dancers should dance. Even if I were the sole developer, I would be inclined to build things this way. I want it to be easy for my client to find someone to work on the application. If they want to change the look and feel, the should not have to hire someone who can write or read SQL statements.

I am attracted to ORM tools like Hibernate, or, in my word-a-day Microsoft centric world, NHibernate. I like SQLAlchemy, and SQLObject, and I agree with the goal they try to achieve. Application developers should be spared the pain and inconvenience of dealing directly with the database. Not all developers are able to build a good database, and not all applications should require them to mess with SQL. Fair enough. Ruby on Rails rocks for certain types of applications, and I can see why it is catching on.

However, I CAN build a good database, and I want to. I specialize in applications that contain many rules, and run against databases that may be required to grow and change over the years. Often, my databases have been used by multiple applications, not just one. I need to know that the rules will be applied well no matter what language is used to write the application. I also need to know that the database will perform well, and I do not want to rely on the poorly written queries the developers often write. (It is not their fault, DBAs do not always write good code.)

Since the dot com meltdown, many of my clients have started to look for the one-stop-shop. They would prefer to hire one developer to do all of the work in order to control the project budget. My experience, as a consultant who often gets asked to work on somebody else’s code, is that the one-stop-shop does not work. Often, the result is a bad database sitting under an inflexible web application that is not standards compliant.

Basically, I like stored procedures, and I am suspicious of frameworks and the magic they perform. In my own free time, I can see myself playing with Ruby on Rails, Turbogears, Hibernate, or some other framework. But, I would want the flexibility to use and take advantage of my database skills. Am I being a fuddy-duddy? I like MVC frameworks, but I still want to be use my database skills.

For a discussion of what some of us experience, see this complaint about Ruby on Rails in the enterprise.

I cannot count the number of times that I have engaged in conversations about the merits of various content managements systems. Does it matter? Perhap it matters, but what matters more is to plan how you are going to manage content in the enterprise. Planning includes defining process and workflow, and determining how to staff positions on a content management team. Planning may even invlove defining what you mean by content. For example, there could by FAQs, press releases, internal news, internal blogs, training materials, policies and procedures, staff directories, on-line threaded discussions and so on.

In the begining, it does not matter which system you use. It matters that you can plan the work and establish a budget. Then, choose the system while mapping it to your budget and needs. I have seen many content management projects fail because they were too open-ended, or because they were incorrectly defined. I was once involved in a major initiative by a large fortune 100 company that spent tens of millions of dollars and got nowhere.

Sometimes, I talk to developers who are working on projects that they call ‘implementations’: as in, we are implementing a content management solution using Hummingbird. I will often press these developers to explain the problem they are trying to solve. After all, how can you say that you are implementing a solution if you cannot state the problem? The developers will often say that they need a system that can retrieve both formatted and unformatted content. They will also pepper the discussion with all the right catch phrases and acronyms: XML, faceted metadata, fulltext indexing, and many others.

However, it is obvious to me that the developers are solving their own problems first. It is MY problem as a developer to find a way to search both structured and unstructured content, but users do not care. The user wants to find a press release, a biographical reference or a piece of correspondence. When these gigantic content management projects launch, the users do not like them, or they feel intimidated by them, or they find themselves without the team structures and processes to use the solution. The project fails, and the developers blame the users.

So, ok – I ended up repeating Jeffrey Veen’s main points. I couldn’t help myself.

NOTE: To be fair, there are cases when large organizations, like the Provincial government of Ontario, tries to provide a solution for extremely large groups of users in multiple business units. Sometimes, the idea is: build it and they will come. A lot of good thinking goes into these projects, but I cannot help but thinking that the task is too big. Time will tell. Personally, I do not think one size fits all.

I ran across the following article entitled “Why Good Programmers are Lazy and Dumb”, and I found it interesting. This entry received several comments. My favorite was: “I r lazy.”

My years of programming have shown me that smart people can become their own enemies. For example, I cannot count the number of times I have assigned work to a programmer only to find him sitting there later in the day trying to fugure out why his code does not work. Long after it is obvious that it is time to come up with an alternative solution, the programmer will try to find out why his solution does not work – after all, it SHOULD work. I have to remind the programmer that his job is to provide a working solution – not to understand why a solution does not work. Smart people can become side tracked by their own curiosity.

We all go through a phase of thinking that we are smart. I can remember spending hours ensuring that my code on a particular project was reusable, only to realize years later that I had never reused it and never would.

It turns out that the real skill is knowing when to make code reusable not knowing how to make it reusable. Many programmers think they are smart when they know how to do something – knowing how is only the beginning. Sometimes it pays to ask the stupid questions:

1. Is this framework really necessary?
2. Will I ever use this code again?
3. Am I working to hard to do something that should really be easy?
4. Does doing things the so-called smart way always have to be so hard?

I have used Eclipse and I would not want to use anything else for writing Java code. I hold out great hope for this project, and I look forward to evaluating many of the other tools that plug into this framework. I have used PyDev to edit Python code in Eclipse, as well as Red Robin for Jython. (At the time, I was not happy with auto completion, but I was impressed enough overall.) I use Quanta to edit PHP, and I am happy to do so. I would like to find a good open source JavaScript editor that performs good auto completion – using I rely on auto completion for the DOM.

Since I jump back and forth between Python, Java, SQL, HTML, CSS, JavaScript, C, C#, PHP, and VB.NET, I often do not remember method names, but I recognize them. For example, today I am working in C# and I am using a datareader to retrieve data from the database. I know that my datareader can be populated by executing a method on the command object, but I cannot remember if the method is called ‘ExecuteReader’ or ExecuteDataReader’. Auto completion helps me be productive as I type rather than requiring me to reach for a manual or use a help file. I am not ashamed of relying on the auto completion feature of the IDE.

I am very happy with MySQL, Postgres and SQLite, each of which I would use under different circumstances, but I need a good modeling tool. I have seen some commercial software that allows for modeling both Postgres and MySQL databases. Once a database reaches a certain size, I like having a picture. I also find that it is useful to sit down with a client to validate the physical model against the business rules. I find that clients can read an ER diagram if I am there to help. Therefore, I will be shopping for a good tool. I have used DBDesigner for MySQL, and I have enjoyed, but I am getting tired of waiting to see the final version.

I like UML, especially as I am planning. I do not want to pay for Rational Rose. I know that there are modeling tools that plug into Eclipse. I like pictures of both the database and the libraries I build. I want to find a good tool soon. I do not need use cases, and I would not use Rational Rose or even Visio to manage them. Last summer, I was exposed to a product called Catalyze by Steeltrace. Given the way I think and work, no other tool compares. I want to write more about this tool later. This tool generates Word documents, process flows, and uses cases.

I have reservations about the MS Windows bias of the Catalyze tool, but I seem to recall that it supports MySQL. It may not be a stretch for it to support Open Office. I know that Catalyze uses Velocity – if Catalyze uses XML to create the Word document, it should possible to open it with Open Office. This tool is so good, I would be willing to run Windows on at least one of my computers just to be able to use it.

So, must have features for me are auto completion and useful wizards in an IDE, good modeling tools for databases and libraries, and good requirements tools. Eclipse supports some modeling tools, but I have not tried them yet. At the end of the day, I am not so ideological that I would refuse to buy and use commercial software if an open source equivalent is not available. I draw the line at purchasing tools that would require my clients to buy licenses, too. I do not want to pass on costs to my clients that way.

Therefore, I am big supporter of good-enough-technology. There are plenty of vanilla flavored but mission critical applications out there that do not require the most feature-rich tools. Some of us live in skyscrapers, but the rest of us live in houses. Construction techniques that are good for one type of structure are inappropriate for the other.

The question is not whether .NET is superior to PHP, or whether MySQL is superior to PostresSQL. It starts with the requirements, stupid! I feel myself getting cranky every time I read that a certain tool or approach is superior to another.

I assert: certain developers are superior to others regardless of the tool. Also, certain tools are superior to others in certain circumstances. In many circumstances, given the requirements, good enough is good enough!

Speaking of good enough, I am very impressed with SQLite. I find that it is a blindingly fast little database that can be used from PHP, Python, Java and C#. In PHP and Python, I was able to write my own functions anbd use them in SQL statements. If you have a client who does not require heavy support for concurrent database writing, if you are building a desktop application that is being accessed by a single user or if you need a good and reliable in-memory database, SQLite is more than good enough.

I have not put this database through its paces to see how well it runs on a website, but, given its speed, I expect that it releases resources quickly. As long as the web application does not require a lot of writing, I expect the SQLite could support many of the small website and web applications I have seen. Give it a try, and let me what you think.

1. Data is recorded.
2. Data is managed
3. Data is used
4. Data dies

It is possible to be a good “technical” DBA and to build inadequate databases. Many of the textbooks you will see concentrate on mind numbing theory, Codd’s rules etc. This theory is necessary, but it is not sufficient if you want to build good databases. I will assume that you know your basic theory, that you know about first, second, third and fourth normal form; and I will assume that you know how to read an ER diagram. My goal here is not to rehash what every database professional should already know, but to outline a commonsense approach to data that can be layered on top of this merely technical knowledge.

The Wrong Way To Start

I cannot count the number of times a client has presented me with a list of fields that was prepared by a business analyst. The list usually included data types for each field, and it may also have included rules that apply to each field. In each case, the client expected me to accept this list and to start modeling a database right away. This is wrong! Wrong! Wrong! And, I will tell you why. I cannot list all the reasons, but I will list the few that correspond to my own personal best practices and ways of working.

To be frank, I have never found the list of fields to be correct. I often find fields omitted, and I also find that some of the specified fields are unnecessary. Over time, I also find that some of the fields in the dictionary are poorly named – the rest of the enterprise is often in the habit of referring to some of the fields the business analyst provided by completely different names. Listing fields is the wrong way to start. It often turns out that the list was assembled by a committee – it is useful, but you owe it to your client to question it. All I can do at this stage is outline the practices I employ to determine the correctness of the so-called data dictionary.

Best Practice A: Paperwork

First, I ask to see the paperwork that corresponds to the fields in the dictionary. I have worked on a few HR applications, for example. Often, each new employee is required to fill out forms. In social work or in employee assistance, these might be called intake forms. I ask to see these forms for three data related reasons: 1) Field names in the paperwork should correspond to the field names in the data dictionary. 2) All fields in the paperwork should appear in the data dictionary. 3) All fields in the dictionary should appear in a forms – if they do not, find out where this data originates. If the field in the dictionary does not have an origin, then perhaps it does not belong in the database.

Seeing the forms will also help you understand the work that the database is expected to enable. Work takes place over time, and time is an important dimension of the data.

Seeing the paperwork will also help you plan the data input screens – fields should appear in the same order on the screen as they do in the paperwork. Also, the application should make it easy for a business user to redesign the data entry fields if the paperwork changes. (Nobody has ever consented to pay me to build such a feature, but I still think it is a good idea.)

Best Practice B: Understanding Each Field

You would be surprised to find out how much data is collected and never used. The rule of thumb is, each field should have a use. There may be statutory or regulatory reasons, or there may be reporting reasons. Why collect information if it is never used in a report, or if it is never used to inform a decision? At this stage, many of my clients have agreed to drop a field or three from the data dictionary.

In brief, each field must have a use or a place in ever stage of the information life cycle. All data must be collected, managed, used and expired. If not, why is it there?

Best Practice C: Time Is A dimension of the Data

Almost every system I have worked on, except for databases I have designed, has been designed to statically represent the existing enterprise. In these databases, departments never merge, business units are never renamed, and business rules are absolute.

I once worked on a billing application where the rules changed each time a contract was renegotiated. Services that were billed as a lump sum in one period were billed by the hour in the next. What made it even more difficult was that a client could renegotiate a billing rule at any time during the life of the contract, and the application was required to generate reports across arbitrary periods of time during which the billing rules could change a few times.

The application I am currently working on contains a table of government departments. Over time, departments may merge, split or be eliminated altogether. A good database will reflect this reality. Departments exist from a certain date to a certain date, at which point they may be renamed, split or merge with another department. The data has to be stored such that reports for previous fiscal years can still be generated using the previous department names, just as reports for this fiscal year must reflect the current department names.

I worked on another billing application where the costs of a project in the database were being shared by different lines of business. The cost share was represented as a percentage in a cost share table. This was all fine and good. The reports were correct every month, even though the cost share was changed a few times over the first quarter. The accountants noticed a problem as soon as they generated the first quarterly report. The amounts billed to each line of business in the quarter did not equal the sum of the amount they were billed each month in the previous monthly reports. (If you can follow that, you are not doing badly.) The solution was to alter the cost share table to reflect the fact that a cost share was in effect from a specified date to another specified date, and the reports had to be changed to correctly charge for services across these periods of time.

It is a good practice to ask your client about the impact time has on each field in the database. You may be surprised by the answers you get. Basically, billing codes and work codes change over time. If there is a requirement to be able to generate historical reports, you have to think about how to handle this.

Best Practice D: Be A Pain

I advise you to be a pain with my tongue planted firmly in my cheek. People skills count for a lot. There is no point in pissing someone off, especially if you need his or her cooperation for the duration of your work. However, you should be committed to doing the job you were hired to do, and you should have a mantra of sorts. Mine is: You cannot pay me to do a bad job. I want to do my best, and I stand up for quality.

A few times, I have been in the position of building a web application on top of a database that was being designed by a DBA. Oracle DBAs seem to feel that their superior grasp of the features and capabilities of Oracle qualify them to build the database. Every product or technology has a culture, and Oracle has a culture of arrogance and control, in my opinion. To the arrogant DBA I would say: Being able to spell does not make you Shakespeare.

You owe it to your client to challenge the DBA to address the important issues above. Do not argue for the sake of arguing, but show him or her that you are creating requirement documents and you need his or her help. You are accountable, especially in these days of one-stop-shopping where the developer and the analyst are often the same person. Hold the DBA to the same high standards you try to hold yourself.

The other issue DBAs, especially old-time Oracle die hards, do not understand is that reports must run fast. Those are the expectations people bring to the web, but, also, web applications will timeout if a report takes too long. Parts of the application may also be stored in the database – these must be retrieved quickly, too.

That’s it for now. I could write more, but this covers some of the broad issues and describes some of the important commonsense approaches I use when working with a client to build a database. In my next entry, I will describe the second stage of the information management life cycle, and how understanding it is important if you want to build good databases.

In the 1990s, my friend Mike Lightheart introduced me to what he called “The Information Management Lifecycle.” In recent years, I have seen the term used by Computer Associates and other companies. I want to introduce the concept as I use it in my day-to-day work.

Data in the enterprise goes through the following stages:

1. It is recorded.
2. It is managed
3. It is used
4. It dies

In the requiremenents phase of a project, it is typical to compile a data dictionary of sorts – a list of the fields that need to be managed in a database. At some point, a data model is constructed, the database is built and the application is developed. I use the ILM to help me provide my client with solutions that I believe are superior to many of the solutions I have seen – if I can be permitted to be so humble.

Over the next few entries, I will outline how awareness of the Information Management Lifecycle influences my design of an application and the underlying database.