While this is something that depending on the complexity and scalability needs of your site one may consider over kill, it is something that takes minimal effort in design to create a system that is innately more scalable. What happens when the data you are querying starts to push out is 5-10Mb of Data to a CSV file?

Option 1

So rather than generate the list of Foo into a CSV at each request why not pre generate the list at pre configured intervals depending on the volatility of the data and save the resultant CSV to a san that can be served up via IIS. In this scenario there is no need for the ASP.NET worker process to get involved and the IIS process can do what it was meant to do serve content.

Advantages

• No need to hit the DB to get the data
• You don’t have to involve the asp.net worker process and let IIS do what its designed to do

Disadvantages

• Your data may be stale up to a preconfigured amount of time (but since its going to a report of sorts it is more than likely stale)

Option 2

So your business requirement absolutely dictates that it is required to be generated and sent as the user requests it. There is no choice, you have but to send it through the response stream. BUT there is another way. Say when this request for the data to be exported VIA an AJAX call to your server the response from the server is a URL to where the file will be output. Now the server it self can then send a message to the export to csv service and request the file be output to the san. The client can then poll the server waiting for a 200 response code, when it does get one it can then redirect the browser to the given URL.

Advantages

• Gives you more or less up to the minute accurate data
• Reduces the amount of work the asp.net worker process has to perform

Disadvantages

• Introduces polling into the design of the client

When designing systems that do not take Scalability into account from the ground up, you generally are missing something. Considering Scalability does not mean that you choose either of these two options all the time. At times it might make sense to think about using the approach suggested in Scott’s article, the key being to always think about know your data and make informed decisions.

Share/Save

Command Query Responsibility Segregation – Udi Dahan from Mark Harris on Vimeo.

Share/Save

The first thing that went through our minds was that we had written the persister wrong, however a persistent Simon continued to work through the issues. Along the way we discovered a few things.

1. NHibernate is faster and therefore able to handle the load better.
2. Proper Saga Design is important.

NHibernate is faster

Most of you already know this but the key thing here is that because its faster it fails less therefore the issues that we saw in the development of the persister was difficult to isolate.

In the Manufacturing Sample that comes with NServiceBus start up all the required services and when the partner is ready run it in simulate mode, it does not fail. doing the same thing when using the L2S persister had the same effect.

Once the number of worker threads on the Order Service Host was increased to 4 could use we started to see a whole bunch of errors. From messages not being completed to violation of primary keys and deadlocks. Using the NHibernate persister required us to increase the workers to 6 or 7 you would start to notice similar errors.This leads me to the next item.

Proper Saga Design is Important

If your not sure about what a Saga is please read the NServiceBus wiki. When considering how you build a system  though that goes into how you would like to handle long running transactions is important. Knowing how many transactions per second. What are the SLA’s? My rule of thumb the more i think about it is that a Saga message that starts the saga should be unique any subsequent messages that are part of that saga should be dealt with a more explicit message handler. So you have a CreateOrderMessage that starts a saga and subsequent messages are handled by UpdateOrderMessages for changes to the order, with CancelOrderMessages and AuthorizeOrderMessages being explicit to the saga.

In my opinion doing so produces both a simpler and more explicit design of your long lived transaction. This however is now going down the path of a different post one that warrants a post of it own.

The Benefits of Reading code became clear the deeper we went through the NServiceBus code base, the more you understand how each piece interacts with the other to provide a great framework. Despite all the hurdles L2S provided it was worth the effort. What must be said for NServiceBus is that it is highly extensible already evidenced by its existing OOB support for 4 IOC frameworks, now 3 ORM’s and in the future with version 2.1 it will support a whole raft of Queued Transports.

Share/Save

This post has been sitting in my drafts for a while. After my day today i felt that it was best that i put my thoughts about service broker down.
In a project that i worked on about 2 years ago Service Broker was used as part of the system to perform data validation.  If success was measured by the amount of time that a system has been in production without major issues then i can say that it was successful.

But if a system was to be judged by how much of a pain it is to maintain then l must say that its was not a glowing success. Below is a list of things that i found as good or bad about it.

Bad things

1. Limited information on the internet
2. Testing is extremely hard (compromises everywhere)
3. Troubleshooting is a pain
4. It could stop working and you may never know, (if using a fire and forget mechanism)
5. Poison messages disable queues (would have been nicer if it moved the message to a error queue)

Good things

1. Its Asynchronous
2. Its Asynchronous
3. Its Asynchronous

Yes this list is one sided but true. The question that is important though is

Would i use Service Broker again?

The answer to that question is strangely YES. Only in very specific circumstances where system is ok with not having some data loss because of queue being disabled

What would I use instead?

NServiceBus of course. It provides an answer to all the negatives of service broker and is still asynchronous.

Share/Save

Finally this week I have started to read ADDP and have been struggling to put it down to do anything else. IMHO this is one of the best Technical books ever written. As a newbie to DDD there is no better starting point than with this book.

Jimmy’s style of writing is very easy to read and understand. The content of course is brilliant, I have just finished reading chapter 2 and feel a few years wiser.

My advice to any developer looking for enlightenment is to read Jimmy’s book. Perfecting the use of Domain Driven Design is not something that comes easy. Even in the smallest application the value of using it is immense. I have a long way to go to become a good developer, and reading this book is going to be a big help in that process.

Share/Save

I guess when PDC rolls around in a few weeks and more news around what Oslo is and when the bits roll out to developers and architects to see, touch, feel and smell what the CSD and Dev Division guys have come up with is when the real questions need to be asked. All well and good to speculate, I just hope that some parts of the Oslo effort will touch on some of my desires.

UPDATE: Mike Taulty has posted links to a whitepaper and a screencast that explain more details about .net 4.0

Share/Save

I imagine it to be a tool that enables these Information Workers to use some of Oslo’s features much like using Visio, but instead of static elements there are dynamic objects drawn from the repository that perform a specific business purpose, when put together by the domain expert enables them to perform a task that would have required a developer to perform otherwise.

As it turns out I may not be that far from the truth, or some part of it. Ron Jacobs interviewed David Chappell at Tech.Ed just past and in that conversation David divulges a few interesting bit of information.

Two things that caught my attention was the “Lifecycle Manager” and the “Process Server”.  But here are my two predictions about what these two things could be. (Yes I am going out on a limb)

The Lifecycle Manager – A long running workflow or Saga manager.

The Process Server – A Service Bus implementation.

The video  of Ron Jacobs and David Chappell along with David’s loose timeline/ roadmap to Oslo is after the Jump

endpoint.tv – The Road to "Oslo"

Thanks to Lars Corneliussen for the following

David Chappel also described the release roadmap, but without confirming any exact dates:

1. A CTP Release around PDC (October 27?)
   He didn’t say what that will include.
2. New Workflow Foundation with .NET 4 and a new Visual Studio version
3. The repository including the modeling language and its visual editor + a first part of the process server
4. A more complete version of the process server, including lifecycle manager (whatever that means)

There is nothing left but waiting for the PDC in October. So see you there!

 

Share/Save

With .net 4 they are not touting it to be a additive release like 3.0 and 3.5.  Wenlog says that 4.0 will be the next major release since 2.0,

• Major improvements for WCF and WF
• Built in dynamic language support
• adding support for a new language (dynamic) for Oslo’s “model driven” programming

But to get the full picture we are going to have to wait for PDC, when the first CTP bits should be available. I can only predict exciting times ahead, as always more things to learn, I guess its time to open Robert Pickering’s book Foundations of F#

Share/Save

• Biztalk V Next
• Emacs.net
• Microsoft ESB
• All of the above and more…

IMHO I believe it to be everything and more, but what proof did i have to support it? Well Doug Purdy announced on his blog about hiring for Emacs.net This raises the question of why another IDE/Text Editor when Visual Studio is very powerful, unless you have tried working with XAML. I digress.

Last night there was posts from Doug Purdy and Don Box that began to unveil bits of what make Oslo.

Doug Describes it as three simple things.

• A tool that helps people define and interact with models in a rich and visual manner
• A language that helps people create and use textual domain-specific languages and data models
• A relational repository that makes models available to both tools and platform components

and Don says that Olso is being created with a couple of intentions

1. We’re making it easier for people to write things down in ways that make sense for the domain they are working in – the common term for this in the wild is modeling.

2. We’re making the things people wrote down accessible to platform components during program execution.

Based on this I would take a punt and say that Oslo is going to create a platform for developers to create DSL’s that will be stored in a repository, this DSL is then retrieved and consumed by a visual tool to create on the fly mash-ups of their domain. The key part here is that the visual tool will be something that can be used by Information Workers, Business Analysts, and Project Consultants.

I imagine it to be a tool that enables these Information Workers to use some of Oslo’s features much like using Visio, but instead of static elements there are dynamic objects drawn from the repository that perform a specific business purpose, when put together by the domain expert enables them to perform a task that would have required a developer to perform otherwise.

This is something that is peeking my interest. Wow I can imagine every medium to large business needing something like this. I cant wait for PDC anymore, I just wish I could be there, then again that’s what a Webcast is for.

Doug calls this the “end of the beginning” of his vision of enabling everyone to be a programmer if they can pull something like this of then he may be correct. But I will wait and hold my judgement till I can play with the Bits that should be available at PDC08

Share/Save

Most samples I have seen demonstrate communicating within a single database, and hence use the SqlTransaction Class The code below is an example of sending a message to a SSB Service  in a SqlTransaction.

   1: private void SendMsgToSSB()

   2: {

   3:     SqlConnection conn = null;

   4:     SqlTransaction tran = null;

   5:     Conversation dialog = null;

   6:     Service client = null;

   7:     string connString = "Persist Security Info = False; Integrated Security = True; Initial Catalog = MyServiceBrokerDB; Data Source = .; Connect Timeout = 30;"

   8:     try

   9:     {

  10:         conn = new SqlConnection(connString);

  11:         conn.Open();

  12:  

  13:         // Begin a transaction

  14:         tran = conn.BeginTransaction();

  15:  

  16:         // Create a service object

  17:         client = new Service("MyCoolService", conn, tran);

  18:         client.FetchSize = 1;

  19:  

  20:         // Begin a dialog with the MyCoolService

  21:         dialog = client.BeginDialog(

  22:             "MyCoolServiceTarget", null, "DEFAULT",

  23:             TimeSpan.FromMinutes(1), false, conn, tran);

  24:  

  25:         // Create request message

  26:         string outgoingBody = "my really cool msg that broker understands";

  27:         Message request = new Message("DEFAULT",

  28:                                       new MemoryStream(Encoding.UTF8.GetBytes(outgoingBody)));

  29:  

  30:         // Send the message to the service

  31:         dialog.Send(request, conn, tran);

  32:  

  33:         dialog.End(conn, tran);

  34:         tran.Commit(); 

  35:     }

  36:     catch (ServiceException svcEx)

  37:     {//deal with this pesky service exceptions

  38:     }

  39:     finally

  40:     {//clean up here close connections etc.

  41:     }

  42: }

But in scenarios where the transaction needs to be elevated from a lightweight transaction to one that needs the aid of the DTC the SqlTransaction object is not going to cut it for you.

With a simple change (in bold and italicized) you can then support enlisting in a Distributed Transaction.

   1: public void SendMsgToSSB()

   2: {

   3:     SqlConnection conn = null;

   4:     Conversation dialog = null;

   5:     Service client = null;

   6:     string ConnectionString = "Persist Security Info = False; Integrated Security = True; Initial Catalog = MyServiceBrokerDB; Data Source = .; Connect Timeout = 30;";

   7:     try

   8:     {

   9:         conn = new SqlConnection(ConnectionString);

  10:         if (conn.State != ConnectionState.Open)

  11:         {

  12:             conn.Open();

  13:         }

  14:         conn.EnlistTransaction(Transaction.Current);

  15:  

  16:         // Create a service object

  17:         client = new Service("MyCoolService", conn, null);

  18:         client.FetchSize = 1;

  19:  

  20:         // Begin a dialog with the MyCoolService

  21:         dialog = client.BeginDialog(

  22:             "MyCoolServiceTarget", null, "DEFAULT",

  23:             TimeSpan.FromMinutes(1), false, conn, null);

  24:  

  25:         // Create request message

  26:         string outgoingBody = "my really cool msg that broker understands";

  27:         Message request = new Message("DEFAULT",

  28:                                       new MemoryStream(Encoding.UTF8.GetBytes(outgoingBody)));

  29:  

  30:         dialog.Send(request, conn, null);

  31:         dialog.End(conn, null);

  32:     }

  33:     catch (ServiceException ex)

  34:     { //deal with the exception

  35:     }

  36:     finally

  37:     { //clean up

  38:     }

  39: }

Again this code has a small issue what about if there is no Transaction available then sending a message without a transaction could be a bad thing. With one more change its ready to support both scenarios.

   1: public void SendMsgToSSB()

   2:   {   SqlTransaction tran = null;

   3:       SqlConnection conn = null;

   4:       Conversation dialog = null;

   5:       Service client = null;

   6:       string ConnectionString = "Persist Security Info = False; Integrated Security = True; Initial Catalog = MyServiceBrokerDB; Data Source = .; Connect Timeout = 30;";

   7:       try

   8:       {

   9:           conn = new SqlConnection(ConnectionString);

  10:           if (conn.State != ConnectionState.Open)

  11:           {

  12:               conn.Open();

  13:           }

  14:           if (Transaction.Current != null)

  15:           {

  16:               conn.EnlistTransaction(Transaction.Current);

  17:           }

  18:           else

  19:           {

  20:               tran = conn.BeginTransaction();

  21:           }

  22:  

  23:           // Create a service object

  24:           client = new Service("MyCoolService", conn, null);

  25:           client.FetchSize = 1;

  26:  

  27:           // Begin a dialog with the MyCoolService

  28:           dialog = client.BeginDialog(

  29:               "MyCoolServiceTarget", null, "DEFAULT",

  30:               TimeSpan.FromMinutes(1), false, conn, null);

  31:  

  32:           // Create request message

  33:           string outgoingBody = "my really cool msg that broker understands";

  34:           Message request = new Message("DEFAULT",

  35:                                         new MemoryStream(Encoding.UTF8.GetBytes(outgoingBody)));

  36:  

  37:           dialog.Send(request, conn, null);

  38:           dialog.End(conn, null);

  39:           if (tran != null)

  40:           {

  41:               tran.Commit();

  42:           }

  43:       }

  44:       catch (ServiceException ex)

  45:       { //deal with the exception

  46:       }

  47:       finally

  48:       { //clean up 

  49:       }

  50:   }

I have also been making a few changes to the interface that I intend to publish shortly.

Share/Save