The regular readers of this blog know that my database modeling book was cancelled when Microsoft pulled the rug out from under M and the repository. I did a lot of good writing about modeling in general, so I wanted to put some of it up here on the blog, since the cancellation is official now. This is the second part of Chapter 1.

Why modeling?

Much software is designed on napkins. A story, a mockup, a model is really all that you need to design how a piece of software works.

This book isn’t about user stories or screen mockups. It is about models, and modeling models. It is one part of the trifecta of software design, and it is a very important part.

What is a model? When you say model to my son, he thinks Legos. That’s not far from the truth. A model is the whole of the component parts that make up software. In the example in Chapter one, the model consisted of trucks, boxes, and the relationship between the two.

The further you get into the so called ‘enterprise’ world (meaning really really big software) the more having a consistent model makes it possible to design intelligent software. This is because a good model brings consistent terminology, an abstraction of ideas and an understandable message to the party.

Creation of a consistent terminology

Working on a model makes you decide what to call things in your domain. In the example in Chapter 1, Boxes could be called Containers. They aren’t. They are called Boxes.

Purists will say that it isn’t just the act of deciding what to call things that matters, but instead the decisions that are made. I disagree on matters of principle here. The simple act of deciding to call something a box and not a container is important. Discussion on what a container is matters. Later on, might you have a bucket? Should boxes and buckets both be containers?

This conversation is frustrating but important. This is the selection of nouns in your new language, and nouns matter. They participate in a common language for your business, both in everyday discussion and in software development

Language and software design

They are called programming ‘languages’ for a reason. Spoken languages have nouns and verbs, as do programming languages. The difference is that precision matters a lot more in programming languages, so the ‘dictionary’ of nouns needs to be exactly accurate.

In programming languages, you define your nouns and verbs. Nouns are class instances, and verbs are methods. In a language like Visual Basic, you define the members of the language as you go:

Public Class Box

Public String Destination

End Class

The system has a concept of a box, and the box has a concept of a destination. We didn’t call the box a container, we called it a box. The box has a destination. This is where the box is going. We didn’t call it an endpoint. An endpoint might mean something else.

Verbs work the same way. You might be able to route a box, for instance.

Public Class Box

Public Destination As String

Public Function Route() As Boolean

Return True

End Function

End Class

Now the Box knows a verb – you can route a box. It is an action word – something you can do. We are literally building a new language here – the hip term is a Domain Specific Language, or DSL. It is not a general language. It is specific to our needs.

There is a problem here, though. The Visual Basic code is something that is only used by the developers. Left to their own devices, programmers will come up with the language to describe your business all on their own, and then it won’t be a common language, used by the business and the IT staff.

The key is in a common language

The reason that domain languages are important to modeling is that the software comes out a lot better if both the business users and the software developers use the same words. I probably don’t even have to mention this, because most all of the fine readers of this book have had that conversation:

“But that process only occurs if the router has that one form.”

“Which form?”

“Oh, I don’t know. That one form with the routing information. “

“Which routing information?”

“I’m not sure. It’s a little different every time.”

A common language mitigates this conversation, because they know to say “destination” and not “that information that is needed for the route.” Things have a name. Those names reduce confusion, and cause better software development.

Description of ideas in an abstraction

Creation of a model provides an abstract space to work with the ideas in the system. The architect and the business analyst don’t have to depend on concrete examples with exceptions left and right to design the software, they can work in generalities.

Boxes can be routed, for example. The development team doesn’t have to deal with the fact that Box ABC123 went to Cleveland, and Box DEF456 went to Cincinnati. The exceptions in specific cases are important, and need to be modeled in their own right, but they don’t need to be in the general abstraction that is used to talk about the system in at the 1,000 foot view.

Making models understandable

Abstraction is important when talking to users. To define user stories, the business analyst needs to talk to users all along the process tree. Most users don’t have visibility into the whole process. The receiving guy doesn’t know how Box ABS123 got to Cleveland, but he probably understands that the box was routed. Even if he doesn’t, this idea can be explained, because it is understandable.

Development teams get entrenched in detail. This detail is a necessary as the software gets developed. With an understandable model, though, there is always somewhere to go when you need to return to ‘home base’ and cover the big picture again.

Images versus text

Usually, architects develop diagrams to show models, like Figure 1-4. This is fine, but it gets to be far to complex after the details is added. Few diagrams can be easily searched, organized or simplified. Since we are looking for an abstraction, simplicity is king.

Figure 1-4: A model diagram

An option for a model is to eschew the boxes and use text. Models developed in text can be more easily simplified, and are much more searchable.

Text has a lot of benefits when you are creating an abstraction. First, the viewer doesn’t get tied up by the lines. Often, relationships between items (represented by the lines) are far too complex to make a readable diagram. If you have a simple list of the items that are related, you can just read them rather than tracing them.

The other benefit of using text for the abstraction is that there are fewer blocks to broadening the abstraction. When you realize that the box actually has a relationship with a product, you can just add it, rather than finding the product box on the other side of the page and finding a place for the line.

The conversation between the architect and the developer

The third consideration of creating an abstraction is the conversation amongst the development team members. The problem domain is rarely known when a group of developers get together to solve a particular business problem. Usually a new feature is just that – new – and the terminology being used is foreign to all but the business people directly involved with the process.

Use of a model mitigates this issue by firming up the terminology right from the start. If all of the entities in the business domain are defined and named, the conversation can revolve around the abstraction rather than examples. Even if the business unit itself uses mixed terminology for parts of the business domain, the model will sort out the communication between the architect and the developers.

This conversation is rarely called out, but it a very important conversation indeed. In small and large projects, the big picture is usually in the architects head rather than written down. The architect tries to pass the feature specific information to the developer, shielding them from the big picture. The developer tries to build a focused feature without insight into the overall system. Hilarity ensues.

The construction of a well-understood model provides an abstraction that the architects can use to give the developers context. Context is very important in system development. It doesn’t get in the way of the detailed development work that the programmers are completing, but it does lead to the discovery of contextual errors. Developers can find problems that relate to the system as a whole and apply the fix holistically, rather than just locally in their own feature.

Forming an understandable message

Creating a model allows for communication to be constructed that actually makes sense – especially as it relates to change. In order to communicate to the users and developers about a system, it is necessary to clearly describe the focus points of the system. To clearly describe anything about the system, it is necessary to start with a model.

To some extent, this pales with the need to provide the next guy with a path. Documentation of systems at the design level is notoriously bad. It is never up to date, practically never complete. It is confusing and passes on no knowledge of the business domain. It is useless. Software modeling – done correctly – mitigates this considerably.

Communicating to the user

The user is the person for whom the software is written. It is important that the development team can communicate on a few different levels. First, the user needs to be able to pass on business functionality to the architect. Second, the development team needs to confirm functionality after consuming it into the rest of the system.

During the initial communication about the system, the first thing that needs to be completed is a software model. The entities in the model will provide a base level of communication about the rest of the software. As functionality is discussed, there will be no need to focus on specifics because the model is specifically understood, not just generally assumed.

As important is the review of software designs. When user stories and mockups are reviewed with the user prior to commencing development, there is an understood terminology behind the discussion.

Communicating to the developer

The conversation between the architect and the developer was discussed above, but it bears mention again in this context. Creation of a context for the developer to create features makes a huge difference.

Context isn’t the only issue, however. Users communicate with developers too - especially in agile environments. Testers communicate with developers. Managers talk to everyone. Wouldn’t it be nice if everyone could just use the same terminology? I certainly think so.

The model brings an accurate terminology to the whole team, especially when communicating with the developer. As with the other examples in this section, the terminology reduces errors and the time it takes to communicate ideas.

Communicating to the future

Probably the largest afterthought in system documentation is the next guy. After all, the development staff isn’t being paid to make version two easier, are they? They are being paid to write version one. Nonetheless, having to edit a system is a necessity. Either the technology will move on and the update wizard will come calling, or the business rules will change.

The best way to improve on systems documentation is to remove the problem of updating it on a regular basis. A quality software model will assist with that because it will update as the code is updated. Working from a model removes some of the need for comprehensive documentation.

The business case

It is tough enough to explain to the CIO why you need to upgrade to TFS 2010. Describing just how important it is to totally change the development methodology of the organization even few years is really a problem.

In Software Language Engineering, Anneke Kleppe points out that we need to upgrade our development methodologies now because it’s the shiny new thing, but because we are doing more with less. Dijkstra (1972) called it the Software Crisis, and it is getting worse.

Taming complexity

Year after year, software developers are asked to tackly more complexity. Kleppe describes an environment where it used to be enough to have ‘Hello World’ show up on the screen. With the advent of GUI, consider all of the technologies that have to be mastered to get it on the screen - CSS, windowing, threading perhaps, markup, the list goes on.

What’s more the additional computing power has led to additional expectations of users. Now you are expected to deliver identity and membership, with ‘Hello Mr. Sempf’. Or even delivering more, localization and personalization, with ‘Good evening Mr. Sempf.’

Modeling software tames complexity. Part of the growing complexity of software is that no one person knows the whole system. It is a common story: features need to be added, but the UI guy is on vacation and now there are 24 business rules in the RDBMS.

Accurate models make for understandable software. While nothing makes software easy, not having an accurate model will certainly make it harder to understand.

Enhancing communication

From the project management perspective, nothing makes life better than communication. Weather agile or waterfall, trust and transparency are key. Both of these characteristics require communication. Communication is hard when half of the people at the table calls a container of items a ‘box’ and half of them call it a ‘carton.’

Modeling software enhances communication. Even if all that is done is the simple act of validating the common language it will make talking about the project a lot easier. I hope that you would take it even further than that.

As a project grows, features are added. To build a system of language that assures that the concepts in the application are referred to the same way throughout time. This is essential to good communication, and ease of updating.

With real software modeling, and a metadata implementation, the model follows the software. This means that throughout the lifecycle, the semantics are an intrinsic part of development through the domain specific languages provided in the model.

It's common knowledge that I have been following Oslo / SQL Server Modeling Services very closely.  I am working on a book on the topic, and have posted a number of blog entries.  The speaking circuit has been good to me too, and I have given my Software Modeling With ASCII talk five or six times already this year.

My focus has been on M, but today we are talking about Quadrant.  Quadrant is part of a trio of tools that includes M (a language to build data and domain models) and Modeling Services (a set of common models and repository).  Quadrant itself is tool to interact visually with SQL Server databases.

I've been watching Quadrant for over a year now, and I had a lot of questions about its viability in the marketplace.  As a data management tool, it was underpowered, but as a data browsing tool, it was overpowered.  When I eventually came to realize that it could be a domain model browser, my interest was piqued.  Since you could define the quadrants using M, it would be effectively possible to build comprehensive data management 'dashboards' in Quadrant, and use them in a power user role.

Over time, however, I began to realize that this was an edge case.  The business users and data managers that need these solutions will still find development in M too time-consuming, and the professional developers who would be asked to help them would just rather work in C#.  It will end up that the business users will go back to Access and Excel, data managers will just use SQL Management Studio, and professional developers will use Windows Forms or XAML in C#.

Apparently, Microsoft saw this too.

I am sad to see Quadrant go.  It was a beautiful application, and could have been a foundation to a number of very, very cool tools.  Hopefully the Data team wil find another use for the technology.

It should be noted that SQL Server Modeling is not necessarily dead at all.  Models can still be built with M, and exposed to the world in OData.  Applications can still be built against this model using Visual Studio, and the data can still be managed using SQL Management Studio.  

The loss of Quadrant doesn't impact this vision, and I hope that Microsoft realizes this and continues down the path toward an enterprise-class repository.  It's the last piece of the puzzle that keeps large enterprises from deploying SQL Server in application-centric environments.

A month or so ago, lockpicker and friend Schuyler Towne and I agreed to push eachother a little to do some video in our respective fields.  Now, I am an accomplished lockpicker, and Schuleyer is a fantastic artist, but his gift is lockpicking, and mine is software development so we stuck with those topics.  We even went so far as to set up a Google Spreadsheet to track our progress.

Schuyler, of course, cheated, and has a video already in the can.  It then took me a month to get around to shooting my first screencast.  It's on using M to model everyday databases for everyday projects:

There is some mobil phone background noise.  Sorry about that.  Lesson learned.  The demo is good though.  It shows just how easy Microsoft is making it to build a complete, source controllable data model and deploy it to SQL Server.  It's pretty slick.

I hope everyone enjoys the screencast.  Next one is on generic collections in C# 4.0.

For a long time, I have used the ‘totally unique’ power of the Guid to design the ‘exactly one of several collections’ problem in my databases. The Zen case is that of the container in an inventory management system.  The container can be on exactly one of the collection of stores, or exactly one of the collection of trucks, or exactly one of the collection of warehouse locations.  I can’t, in any case, be in two locations, or no location.

How I usually implement this is to have a single LocaationId in the Containers table that is of type GUID, and is a foreign key for the TruckId, StoreId or WarehouseLocationId.  Since Guids are globally unique I don’t have the worry about a duplication between tables.  Effectively, the Stores, Trucks and WarehouseLocations tables have an implicit uniqueness constraint, which I can enforce in code if I am getting really squirrely.

I was pleased to see that Entity Framework 4 handles this well.   A “Model from Database” command puts up an entity model that looks strikingly like our domain model diagram.

All I have to do here is alter the navigation properties to show a single location, but that pass back either a store, truck or warehouselocataion, based on maybe a simple Location interface.  Good enough.

Where I was curious is in learning how M will handle this design.  As it turns out, not as well. I ran the “Model from Database” and got this:

module dbo
{
    export WarehouseLocations, Stores, Containers, Trucks;
    WarehouseLocations : {({
        WarehouseLocationID : Guid;
        Aisle : Integer32;
        Shelf : Integer32;
    } where identity WarehouseLocationID)*};
    Stores : {({
        StoreId : Guid;
        StoreNumber : Text where value.Count() <= 16;
    } where identity StoreId)*};
    Containers : {({
        LocationId : {
            StoreId : Guid;
            StoreNumber : Text where value.Count() <= 16;
        } where value in Stores;
        ContainerId : Guid;
        Description : Text where value.Count() <= 128;
    } where identity ContainerId)*};
    Trucks : {({
        TruckId : Guid;
        VehicleCode : Text where value.Count() <= 16;
    } where identity TruckId)*};
}

Yeah, that wasn’t what I was looking for.  Somehow, the members of Store ended up as values of LocationID as an entity …. meh.  It’s CTP software, but it is worthwhile analysis.  Perhaps I’ll dig in and see if I can figure out how M came up with this after the weekend.

Thanks to Mike Wood and others for asking me to give my SQL Modeling talk at CODODN.  Events like CODODN are important, because they bridge the gab between local events and the larger regionals like CodeMash.  Smaller groups sometimes mean better hallway conversations and the like.  Kudos to all those who participated in getting this together.

Anyway, here is the solution (WarehouseManagementSystem.zip (2.65 mb)) from my talk.  No slides for this talk, just a little talking and a lot of coding.  Get the bits from the SQL Modeling Website, and make sure you have SQL Express 2008 installed.

 Thanks to all who attended; good questions and insight.

Thursday night I had the honor of giving my new talk “Software Modeling with ASCII, and no I’m not kidding” to the Columbus Computer society of the IEEE.  They were very welcoming and enjoyed the talk, and had a number of comments about the technology and its use.

I start the talk with what essentially be the first chapter of Professional Software Modeling.  WE cover the problems with current modeling systems, and the timeline for modeling and object/relational mapping

The bulk of the talk is effectively a demo of deploying a database and entity classes form a simple model and then generating an ASP.NET MVC web site from the model.  It is similar to the hands-on-lab at the PDC, with the Mini Nerd Dinner. 

Right off the bat, a listener pointed out that we HAVE these tools already.  Don’t we have XML?  What’s wrong with that?  Tools like WSDL and EDMX solve these problems, and are human readable! Why do we need something else?

I agreed in principle but I pointed out that not many people think that XML is not human readable any more.  The sample code was, but the 150,000 line long files that end up getting used are NOT.  Especially when there are only about 1,500 important lines in the file.

The XML is still there, I assured everyone, and the model was still in EDMX.  M is just a way to work with the model that is a little more succinct than the XML.  Additionally, I pointed out, there are more semantic pieces to M that we hadn’t gone over.  We had done the nouns, but there are verbs too, if you get my drift.

As we went over how an M model looks in the M file, and how the final database and domain classes look, someone asked the obvious question.  It’s the same question that I asked last fall.

“Great!  What do I do with an EXISTING application?”

I don’t have a good answer for that.  Would I like to be able to take an existing application’s database and look at the representative M file?  Yes.  Can I?  I am not sure I can.  That is an open question.

Honestly, I haven’t taken the time to look into the story for existing applications.  Fact is, most application development is adding features to existing applications and I don’t know how M fits into that.  If it is going to be a good modeling tool for existing application there needs to be a reverse-engineering story, and I hope there is.  I mean, you can always make a model from an existing database, but I am not sure that is enough.

The last discussion we had was about potential.  Specifically – what is Microsoft doing to support hte wide adoption of this product?  How about multiple datasources?  What if I need to model an identity system, where there is an Active Directory, an LDS and a database with identity information?  Can I model that in M?

No, I answered.  Right now M is shackled by EDMX, which really only has a provider for SQL Server and Oracle.  It is theoretically possible to enhance the M modeling superstructure to handle a multi-source database, but it isn’t done yet.

In general, everyone loved the talk, and I am looking forward to cleaning it up a bit and giving it a few more times.  Thanks to Jack Freund and the IEEE Computer Society for allowing me to speak!  Hope to see you all again sometime!

In M you can describe values and use MGraph to infer a data model.  for instance, I can tell it a little data, like this:

module TheStates
{
  States  => { "Ohio", "Indiana", "Kentucky" };
}

… and get back the SQL code for my inferred table:

set xact_abort on;
go

begin transaction;
go

set ansi_nulls on;
go

if not exists 
(
    select *
    from [sys].[schemas]
    where [name] = N'TheStates'
)
    execute [sp_executesql] N'create schema [TheStates]';
go

create table [TheStates].[States]
(
    [Item] nvarchar(max) not null
);
go

insert into [TheStates].[States] ([Item])
    values (N'Ohio'),
        (N'Indiana'),
        (N'Kentucky');
go

commit transaction;
go

The problem is with a real world model, say a collection of states that represents a route.  if you try to model that with an extent, like this:

module Proto
{
  Representative =>
  {
    Id => "4";
    Name => "Bill";
    States  => { "Ohio", "Indiana", "Kentucky" };
  } where identity Id;
}

Well, at least it is a Not Yet Implemented error.

untitled3  4,3-8,22    Error   M3999   Not yet implemented: Initializing a collection with an expression that yields a collection (Microsoft.M.SemanticGraph.FromExpressionSymbol)

Still, without something that straightforward implemented, it is tough to call this a modeling language for data.  Hopefully it is in the works soon.

Well, if you went to PDC and did the thing I always do, which is go to every talk that lists Don Box as a speaker, then you probably heard about SQ!L Modeling.  I don’t care about the SQL part – SQL is a database as far as I am concerned – it is an implementation detail.  As a software architect, though, I am very interested in Modeling.

So I get the bits and install them.  Great.  I have Quadrant as a Start Menu item, but Quadrant is Access.  Not interested (for this anyway).  I want to tell Visual Studio how I need my software built.  I need M.

The New Project Dialog has a Oslo Library that might be interesting.  I thought they had nixed Oslo though?  What’s up with that?  Anyway, the description says “A project for creating Oslo Flavored CSharp Library” whatever that means.  I just want to design software, people.  I don’t make ice cream.

I’ll go ahead and model SHARP, my event system, because I know that system well and have been modeling it for years.  To start, I name the Oslo Library SharpLibrary.  The resultant code is unusual, but it is a new language after all.

module SharpLibrary
{
    type Model
    {
        Id : Integer32 => AutoNumber();
        
        Field : Integer32;
        
    }
    
    Modelsamples : {Model*} where identity Id;
    
}

Don’t let the ‘module’ statement fool you – this isn’t a RAD application development language that will remain nameless.  M is a language that will model your middle tier.  Let’s model.  The goal is to build the simple SHARP model, which I used in VB for Dummies and the C# All In One.

I will start with the Conference entity, because it is sorta the middle of the system.  ID is as the example, but I want to add a Title and Description.  First thing is that M needs intellisense.  What’s a string again? String?  Varchar?  Oh, no of course it is Text.  Of course.

More later!