Wednesday, December 18, 2013

Starting you Azure project

Is this your project approach?

Azure Project X == Azure Subscription X
Azure Project X Budget == Azure Billing Alert on Azure subscription X
Azure Project X Monitoring == setup (SCOM) Azure monitoring
Azure Project DEV == prepared to support the application after Go-Live?

Azure Subscriptions
http://blog.kloud.com.au/2013/07/30/good-practices-for-manag…

Azure Billing
http://msdn.microsoft.com/en-us/library/windowsazure/dn47977…

Azure SCOM
http://blogs.technet.com/b/dcaro/archive/2012/05/03/how-to-monitor-your-windows-azure-application-with-system-center-2012-part-2.aspx

Regards,

Sander

Friday, December 13, 2013

The Enterprise Continuum – separation of concerns

There are so many options, and ways of developing a solution that I would like to share some of the guidelines we are developing internally. At this moment, I will do this on the blog you are reading, focusing on the solution architectures. In parallel I’m working for my company on the enterprise architectural guidelines, and I’m trying to follow TOGAF principles to lay down the architecture. I’m hoping to be able to define the architecture context, general architecture, and relate it to the solution architecture.

In the perfect world, with all the time to do this….this should result in Architectural concepts, which will be posted on: http://theenterprisecontinuum.blogspot.nl/ focused from the top-down approach of setting architectural requirements such as ‘every project must leverage a monitoring capability’, and this blog: http://snefs.blogspot.com where I will post the solution for this concept.

The first post (which is basically the same as this):

http://theenterprisecontinuum.blogspot.nl/2013/12/hi-there-what-can-you-expect-on-this.html

 

Cheers,

Sander

Sunday, December 08, 2013

Azure Service Bus – Error handling strategy

At this moment there are several ways to build exciting new applications. In several projects, we are using a hybrid/cloud architecture, specifically Windows Azure. In my upcoming posts I would like to share some of the guidelines we are developing internally, in this case specifically a way of handling errors in Azure queues/topic-subscriptions.

A lot of the Azure (integration) Architectures (and even between web-worker roles) will likely use some elements of the Azure Service Bus, or Azure Queues. Going through the different architectures is not part of this post, so I will suffice with a slide from the Service Bus Deep Dive presentation;

clip_image002[4]

Within our company Caesar, several internal systems have been created and where possible purchased. One of them, CRM4.0 was outdated, or not suited for all our requirements (among them Accessibility online). We decided to migrate our CRM system to the Cloud, using Dynamics CRM. As not all systems are migrate and we are in the process of analyzing the requirements and alternatives, we needed a solution for updating our internal systems which use CRM information.

As Dynamics CRM provides means to push updates to Windows Azure, we have implemented the following solution;

·         Dynamics CRM send Contacts to the Azure Service Bus Topic ‘Contacts’

o   For each system subscription, we have a subscription (e.g. contacts-systemA)

·         Dynamics CRM send Accounts to the Azure Service bus Topic ‘Accounts

·         An internal (windows) Services picks up messages from the subscriptions and sends them to the LOB systems

The following architecture explains this architecture:

clip_image004[4]

 

This worked fine, however, sometimes we had a problem processing messages. After diving into the problem we identified that malformed messages/incomplete accounts/contacts were send, which caused an error, which leaded to the Abandon, the message would remain on the queue, and thus, eventually the problem would occur…..we implemented a maximum number of errors strategy, so ultimately the processing service would stop. Implementing error handling, transient fault handling, and Email Listener did not prevent anything; we did not know when an error would occur and what the error would be.

We stretched the capabilities of the CRM Plugin and CRM configuration which allows you to send all fields, perform validations, however, several things can go wrong:

·         Technically

o   Transient faults – network hick-ups, Azure updates which terminate connections, these call all be handled by implementing the EntLib Transient Fault Handling block

o   Environment Configuration - Azure Topic/Subscriptions have not been created in the environment, these can all be prevented by using a strategy such as proposed in my earlier post

o   Management - Azure Storage account configuration is modified/removed, these risks can be minimized by implementing an solid Azure security policy (and not promoting everybody to co-administrator)

o   Server (processing service) is not available, this should be monitored and causes business issues, but due to the asynchronous setup of this architecture, does  not cause any issues in the system which are not solved when restarting this service

·         Functional error

o   Entity consistency

§  Contacts/Accounts are not valid as not all mandatory fields are set, these can be resolved by managing the CRM Plugin

o   Entity dependencies

§  Contact insert is not processed in the internal system, Contact update will fail

§  Account insert is not processed, relation with account cannot be made, this contact insert will fail

 

Given the problem, some can be solved by implementing readily available frameworks and components, however, for some errors, a strategy is in order. Let’s look at the aforementioned problem in relation to the operations. Processing messages has been implemented earlier by using the peek-lock model where a message is only marked as processed by the following operations on the brokered message:

·         Complete (everything went fine)

·         Abandon (an error occurred while processing)

·         Defer (meta-data can be added to the message, so that the message can be picked up at a later time)

clip_image006[4]

 

Will this solve a functional error? No!

So what we need is a strategy…which allows messages to be stored in a location, related to the queue/topic-subscription, but will not be processed, is ‘dead’, and is queues for further investigation, hence:

“All messages, which cannot be processed, are placed in the DeadLetter queue”

 

clip_image008[4]

 

clip_image010[4]

This will result in the following state:

clip_image012[4]

 

This however, poses several new challenges, what to do with the dead-letter messages, how to restart messages, in the next post I will explain my effort to implement a monitoring solution by using and evaluating several existing frameworks and technologies.

 

To be continued….

 

HTH,

 

Sander Nefs

Sunday, December 01, 2013

Architecture - ISO/IEC/IEEE 42010:2011

After following the IASA Architecture Core course, I like to continue with my personal learning and improvement, and focus on my architectural skills, among others. This year, by following a course on the Theory of Constraints, which is a really interesting theory which will help analyze the core issue behind a problem, and have followed the MetaPlan training which allows for a structured goal oriented brainstorm. For next year, I enrolled in a training on TOGAF. In my preparation for this, I stumbled upon the Open2Study website, where you can follow a lot of courses for free. I enrolled this weekend into the EntrArch course, which includes TOGAF. In one of the additional resources, it referred to a lot of very useful articles.

So after diving into a lot of them, for learning more about architectural styles, frameworks and more. I can recommend the following;

TOGAF

A Comparison of the Top Four Enterprise-Architecture Methodologies

Survey of Architecture Frameworks 

 

Cheers,

Sander