The Tyrant System

Posted on April 9, 2013 by Ben


It’s time to name something in the enterprise integration landscape: the tyrant system.

The tyrant system is where your project dreams go to die. The tyrant system does not answer to anyone. The tyrant system has releases once in a blue moon, and the rest of the organisation had better make sure they fit in. If you don’t like the tyrant system’s API, you are welcome to spend months coding around it, but the tyrant system reserves the right to change and break your code without notice.

The tyrant system gets to say what operating system, what hardware and what other software it will work with. The tyrant system is invulnerable to external change driven by anything smaller than regulatory authorities¹. The tyrant system has a development plan that was set in concrete before you were born; if you want to be agile, knock yourself out.

The tyrant system is easy to spot. It uses 1970′s alphanumeric codes for business terms, and somehow everyone uses them (it is a rite of passage for new joiners to know what a “Z5R” order is). The support team have a web site explaining in great detail what the rules are for the business to get help (and how many months that’s likely to take). In a generally frenetic environment, calm pervades the tyrant system’s development area. There is absolutely no documentation. Recruiters hire for skills in the tyrant system specifically. No one ever explains a tyrant system requirement in business terms. You do it because the tyrant system says it must be done.

There is nothing necessarily wrong with one system to wielding power over others. If you own a business, you want the system closest to the people making money to get highest priority. You have a tyrant when the system calling the shots has nothing to do with that. History offers little comfort. The ruinous reign of a tyrant can last for for decades, and power is never ceded voluntarily. The only hope for us is that, in the very long run, change is inevitable. And perhaps we can also take comfort from the motto of Virginia – sic temper tyrannis!

¹ Assuming your country’s GDP exceeds that of the tyrant system vendor

Posted in Enterprise Integration, Project Management, Vendors | Tagged , | Leave a comment

Rogue Trading for Dummies Part 1 – Using Trade Booking to Beat Risk

Posted on March 20, 2013 by Graham

Have you ever given any deep thought as to how rogue traders manage to get away with their activities for so long? In a trading landscape driven by technology, it should be impossible to get up to anything like rogue trading, shouldn’t it? Well, as we’ve seen in the recent press, it is still possible to get away with it, at least for a while, either until the rogue trader can’t hide it anymore or he brings down your firm. Both outcomes are unpleasant, to say the least.

In this series of articles, we’ll explore one of the shadows where rogue traders hide – behind poor trade booking practices and weak or ineffective policy enforcement. Firms that operate in this way also have to contend with the other two sides of the “I’ve lost control of my bank” triangle, namely poor risk management and inaccurate regulatory reporting. (Want something else?).

So, let’s have a look at why booking policies and enforcement are critical to a firm’s health, where the pain is felt without them and where your rogue trader could be hiding.

Since the severe problems encountered in the financial markets four years ago, government agencies across the globe have accelerated the imposition of increasingly risk averse regulatory reporting frameworks. This has come at a time when firms are still dealing with continually turbulent market conditions and the need to continue to make a profit. It has never been more important to know the risks that your firm is exposed to, to manage them effectively and report upon them diligently whilst still looking to maximise the firm’s assets and resources in the pursuit of that healthy profit.

The question is how do you do this? How do you know what your current risk level is? There are a plethora of risk management systems and strategies in place throughout the industry that will give a clear indication of the risk that the firm is carrying and this supports strategic decision making processes. Obviously, all these systems rely on accurate data in order to ensure that the picture they paint is correct and reflects what’s happening on the shop floor. So the question is, how accurate is the data the flows into your risk management and regulatory reporting processes?

The true answer, no matter how unpalatable it may be, is that years of “just get it done” attitudes to business, system and data challenges has more than likely left you with an inventory of data that, although at first glance may look OK, is in fact misreporting your risk, failing to meet your regulatory requirements, preventing you from leveraging your positions and trading limits to generate more profit and, potentially at the same time, hiding your rogue trader.

To illustrate the point, let’s look at trader A working on a flow rates desk. As well as booking swaps on the Interest Rate Swap template, he has been booking fees on the same Interest Rate Swap template for the past three years, because this is the only way he can get these into the system. Technically, fees should be booked in a different front end system right across the firm. However, neither trader A, nor his extended team, has access to that system. This approach was the result of a poor tactical decision taken in order to expedite the processing of fees on this desk three years ago. The reasons for the decision have now been lost in the sands of time but it works “OK”. Trader A can book his fees and the money pops out at the other end so all must be well.

Unfortunately, this decision was taken without assessing the full impact of the approach and what it would mean to the middle and back office. It turns out that “OK” actually generates significant breaks in the middle office as these fees will not achieve straight through processing. They require manual intervention in order to process them. Additionally, both finance and settlement IT have had to build out and implement tactical fixes at the boundaries of their applications (accounting rules engine and data load and mapping respectively) in order to handle them. These changes have been designed, developed and implemented independantly. They have been built out on different technology platforms by different teams in different time zones. They have been “squeezed” into a heavy book of tactical work in these domains. All three areas:

• are being impacted by poor booking practices in the front office

• have to react to due to a failure to use the correct system for these fees

• are experiencing a daily, ongoing and never ending waste of effort and cash.

Additionally, this poor booking practice (the “fudging” these fees into the trade capture system) has the effect of clouding the firm’s understanding of its true exposure whilst simultaneously impacting its ability to satisfy regulatory reporting requirements.

So it is clear that a tactical decision made years ago, in a pressured situation has resulted in a negative impact on at least three downstream cost centres and may also be causing the bank to submit inaccurate regulatory reporting to the FSA and other regulatory organisations. Given how easily this decision appears to have been made, it is more than likely that many other such tactical decisions have been made across the firm, resulting in other potentially more unpleasant issues in risk and reporting.

Let’s also not forget that this activity could be counting negatively towards trader A’s trading limits. He could potentially be doing significantly more Interest Rate Swap trades. He could be making a healthier profit for the firm. But he can’t as his limits prevent him from doing more than he is, even though a good percentage of his “trades” are actually fees.

All the above problems are a drain on the firm’s resources and have a real cost implication. A “tactical fix” here and a “quick win” there can have major impacts downstream to cost and effort:

The final, logical and disturbing conclusion from tactical decisions that generate tactical system fixes to accommodate them is that these leave the firm open to exploitation. If a trader is aware of the tactical downstream “fixes” that have been implemented in order to deal with poor booking practices, it is not inconceivable that there lies a “window of opportunity” between these tactical patches that could be exploited to hide rogue trading activity. Processes that exclude a trade, because it is deemed to be a tactical fee booking, in order to protect a reconciliation process presents a rogue trader with the opportunity to manipulate this “patchy” control landscape. With the court proceedings currently underway against alleged rogue traders, it is clear that this danger is always present and systems and processes that allow themselves to be “played” only go to encourage this activity.

So, what can be done about this issue? Where do you even start to determine if your data is correct? How do you determine where tactical solutions have impacted on your risk and reporting processes? Over the next few articles, I’ll explore the options available, how they can be implemented and where potential downfalls may hide.

NEXT article: Rogue Trading for Dummies Part 2 – How to waste money on Risk systems

(Want more?)

Posted in Booking Practice, Risk Management, Rogue Trader | Tagged , , , , , , , , , , , | Leave a comment

10 Questions To Ask About An API

Posted on August 6, 2012 by Ben

The presence of an API in the systems we were buying or building used to be a nice feature to have. It would be appreciated by those trying to write management reports, but it would not be foremost in the minds of those making the business decisions.

Today, the API is coming to the foreground. We build our technology solutions out of a mixture of hosted services, vendor systems and inhouse development, and the ability to connect these systems together painlessly is becoming important.  Ask the questions below before you sign off.

1. Is the API documentation publically available?
Assuming there is any documentation at all, this is usually a good measure of how mature an API is. If a technologist is ready to publish their interface to the world, this shows confidence that the interface works, and that they are trying to encourage people to use it. If the API documentation is hidden behind partner logins and NDAs, this shows the supplier aniticpates a higher level of handholding and explanation. Which means more work on your part.

2. Does the API documentation cover messages/methods, fields and values?
API documentation should minimally tell you what messages you can send / methods you can call. Better if each field is described. Best if the allowed values and constraints in each field are  enumerated. There is a special circle of hell reserved for people who write things like “systemGUID: use this field to specify the system GUID.”

3. Does the API documentation cover error cases / codes?
Often neglected in API documentation are the error cases and error codes. I’ve never seen this done perfectly, but a list of numeric codes with explanations of likely causes and remedies is always welcome.

4. Is there an SDK?
The presence of an SDK is another sign of maturity, along with public documentation. A lot depends on the quality here, and you’ll probably have to load it up and spend a few hours using it to find out. Any SDK purporting to be .NET should work in the latest version of Visual Studio without making you jump through project upgrade hoops. Can you make a basic call to the remote API? Is there a test account present for specifically this evaluation purpose? What you are looking for is saving time, and if you’re left feeling “maybe it would be easier to go straight to the database / web services, everyone’s time is being wasted here.

5. Are there end-to-end working code samples?
Although there are a few stateless APIs, most follow the pattern below:
a. Make an authentication request
b. Receive some kind of secret key
c. Make your business requests using the key
d. Sign off

The business request itself may be broken into more than one stage – for instance, if you are updating a contact in a CRM system, you may first need to search for and retrieve an API reference for the contact before being able to udpate it.

So all of this adds up to you having to make a *sequence* of linked calls, which in turn means having some full examples showing you how to do this. But even though it sounds important, most API documentation won’t show them. If you’re lucky, there’s an out-of-date SDK, but chances are you’ll have to phone the company in question.

6. What technology choices does the API support?
This is just a straight question, and depends on your requirements. A SOAP or REST interface is the standard, and will give you plenty of modern choices. Although Amazon Web Services is an interesting example of an API that is ostensibly open (REST), but complex to use, so you are effectively tied to whatever platform you can get an SDK for. (I only know this because I investigated what would be involved in creating an MD5-signed request to AWS in VBA – mon dieu!)

7. Does the API give you a receipt?
Any API should give you some kind of reference to any objects you create or update, and don’t under any circumstances touch one that doesn’t. Better still, you ought to be able to pass in your own reference. The receipt or reference becomes crucial as soon as you want to ’round trip’ an API call. If you don’t get one, you are left with a nasty correlation problem, and end up trying to guess which output record from the API corresponds to your input record.

Whether it’s your own reference, or a system-generated one, make sure you also understand how long those references are good for, and also that you’ve sufficient capacity for your own purposes. (In a trade flow where you might be passing millions of records a day, a ten or thirteen character field is not going to give you much room for comfort.)

8. How is the API metered/throttled?
Per-request is the most common, but you’ll also see bandwidth and storage with modern cloud services. Be very wary of any APIs that are metered per-user, or by the number of simulataneous connections. It’s not that there’s anything *theoretically* wrong with these things, it’s just that if either you or the API-provider don’t clean up old connections properly, you’ll get locked out, which is annoying and hard to debug. I’ve encountered this on two different APIs, and it just makes you want to kill the vendor.

9. Does the API allow reference data updates?
As the systems you target get bigger and bigger, reference data becomes more and more of a challenge, especially when you are integrating System A to System B, rather than User A to System B. You can make User A re-enter information they’ve already entered elsewhere or manually correlate new records to identifiers in System B (as anyone who’s tried to open a new savings account with their bank can attest), but System A and System B are going to need a more resilient method. For a given entity, System A must know System B’s identifier, or how System B correlates to that entity, otherwise you are going to lose referential integrity. In the ideal world, reference entities are managed in one place, and this ‘golden source’ propagates changes to downstream systems throughout the enterprise. If you ever meet anyone who thinks this sounds easy, ask them if they have a global canonical source of country names*.
10. Does the API support asynchronous calls, and if so, how?
For long running transactions, we may well need to tell an API to do something, and then come back some time later to check the results. One way of accomplishing this is with a ‘polling’ architecture. So we make our request, and we are then responsible for monitoring a message queue or a database table for notification of the result. The alternative architecture is to pass in a custom callback on the request side, so that the API itself can call a function when processing is complete. A well known example of this is  PayPal, where you pass them a URL they can ‘ping’ when a payment is completed.

In summary, these are all good questions to ask when evaluating an API. Since the exercise of selecting an API usually happens as part of a wider technology decision, you might not get much time to ask them, or to do much about the answers. Inside that context, I’d focus just on the correlation question. It is a fundamental requirement, and coding around a poor or lazy solution is painful and frustrating.

*Don’t ask this in America. A list of US states and ‘Rest of World’ is not the right answer.

Posted in API, Architecture, Enterprise Integration | Tagged , , | Leave a comment

Business Rules and Engines

Posted on July 31, 2012 by Graham

This topic expands on my previous post “Increasing STP Rates – the Fantastic Four“. Specifically, I’m going to expand on the role of the centralised Rules Engine, what it does and how to use it. (Want something else?).

You’ll no doubt have heard about these before and have perhaps even worked with something like Drools or the Wolf Framework. I’m not going to deliberate the best technology stack here, the best choice for you in that regard will have a lot to do with your current stack, your ability to support that stack and your appetite for 3rd party solutions. Instead, I want to hone in on business rules, how you can classify them and how you can look to implement them.

Our experience has taught us that rules generally fall into a number of predefined classifications:

  • Data handling (completeness, transformation and mapping)
  • Enrichment (internal domain information, allocation, matching)
  • Filtering (inclusion and exclusion)
  • Sequencing (expected order, missing messages, duplication and replay/resubmission)
  • Validation (successes and breaks)
  • Routing (single/multiple destinations, re-routing, replay)

All of these classifications present their own specific challenges. Their correct application is critical to the success of enterprise messaging solutions (our speciality). Regardless of whether you are working on a green-field project or replacing an existing spaghetti middle/back office solution, you need to elicit and classify your business rules correctly.

Let’s run through the classifications and see the challenges that each presents.

Data handling & Enrichment
All messaging systems need to implement these rules. You need to ensure that the message you are processing is correct both syntactically and symantically. Does it conform to your XML schema? Are the required data items present for this business case? Can you consistently map the incoming message format to your domain language? Can you enrich your data to simplify downstream processing?

Ideally, these business rules need to be performed as early as possible – at your system boundary. If you allow malformed and incomplete messages onto your bus via your routing rules, your downstream consumers with feel the pain, your problems will grow expotencially and your costs will increase significantly. Advising your Operations Managers that they will need to throw bodies into the middle office to manually resolve data problems is not a viable career development strategy.

Filtering & Sequencing
Often, your system will be presented with message that are out of sequence, duplicated or that your system does not need to process. Perhaps worse is the scenario where you are not presented with messages that you expect to receive.

You need to understand the conversations that take place in your incoming messages. Do you just receive and process unique, isolated messages or do your messages convey business meaning through a chain and sequence (new booking, reversal, shaped adjustments etc).

Filtering rules will protect your system from data that it isn’t required to process. Sequencing rules will identify missing, out of sequence messages and intelligently manage them until the missing messages arrive and the sequence can be played correctly or until such time as they decide that this is not going to happen and alerts need to be raised.

Validation & Routing
Your rules need to be able to specify the action to be taken upon a successful or unsuccessful outcome. If this message has been handled and enriched; filtered and sequenced and all is well, what do you do with it then? Where does it need to go? Does it need to go to more than one place? What if it has broken one of these rules? What happens next?

The big challenge facing business rules and the rules engines they operate in is that, more often than not, several business rules relating to a single incoming message will be identified. In order to process the message successfully, these rules need to be applied in the correct priority sequence so that business meaning can be achieved. If the first rule fails, then the rules engine will move onto the next rule and will keep going until a match rule has been found. However, be watchful for distinct subsets in these rules that require a hard failure only halfway through the rule stack. Your rules engine must be able to handle this scenario.

 

Whichever way you decide to implement business rules, the above will be invaluable to achieving success in your project. Before I finish, here are just a few parting shots – make sure that:

  • Your business rules are reusable
  • Your rules engine can access external data
  • You have adequate logging
  • You have automated regression testing packs that can verify rule changes quickly
  • You have sufficient granularity in your rules (client/product specific)
  • You have adequate versioning and documentation of your business rule sets

There, I think that just about does it. Business rules are a fantastic direction to take your project in. Just be very careful about how you approach designing and implementing them. And remember, the only thing more expensive than a good rules business analyst is a bad one.

(Want more?)

Posted in Business Rules | Tagged , , , , , , , , , , , , , | Leave a comment

Business Analyst’s Tools – Data Flow Model

Posted on May 30, 2012 by Graham

In this article, I’m going to talk about a simple yet powerful tool that every Business Analyst should have in their arsenal – namely Data Flow Modelling (Want something else?). Specifically, I’m going to talk about producing a tangible asset that can be prodded and poked by the BA team, the development team and even business users in order to gain a clearer understanding of the Target Object Model. I’ve used this approach on numerous occasions, and it is particularly useful in trade flow projects.

Models can take lots of different forms. These range from the vapour model in your stakeholder’s mind to the stack of exquisite UML Use Cases and Sequence Diagrams in your design specification. But the gold standard is a working model that performs processes, uses reference data, makes decisions on a critical subset of data objects and shows a use case being processed through the Target Object Model. We produced one of these on a recent derivatives clearing messaging project, and I’ll use that as an example.

The steps we followed can be summarised as:


Producing the Trade Flow Playlist

The first thing we did was generate a “Play List” of the possible paths our trades could follow through the trade flow. We employed highly sophisticated tools  – some Magic Whiteboard and a stack of Post-it Notes. We drew our messaging domain on the Magic Whiteboard, and then marked out our inputs, our proposed messaging queues and our endpoints (effectively our middle office core applications). We used the Post-it Notes to represent our actors (trades coming from the exchange) and walked through the processing of each message, noting the waypoints they went through, when they got there and what supporting data objects each message required at each of these waypoints.

We were particularly keen to understand how, where and under which circumstances a single actor being input to a waypoint resulted in two actors being output on different message queues. All this was documented on more Magic Whiteboard and used to drive the next process in our model development.

Visualising Use Cases in Visio

At this point, we felt pretty pleased with ourselves. Despite having dry wipe marker on our hands and dirty shirtsleeves, we had a set of scenarios and enough detail to start producing our use cases. We broke out Visio and translated our play lists into use cases, showing the processes our actors would flow through and the data and success/exception/constraints that they would be subjected to. This process gave us our bible of use cases and they were built in such a way as to provide “snap shots” in the life of each actor. These made a useful tool in their own right (having a full set of use cases is important to all projects), and this process also highlighted commonality within some of the processes – allowing specific use cases to be developed for them in their own right (our Adapter Framework being a prime example).

Given the very complex Target Object Model, and the effort required so far in understanding the flow and how it would work in real-time, we wanted something more tangible – something we could run our use cases through and iron out issues, drive out data problems and “taste” the effects of our proposed asynchronous messaging design.

Bringing it alive with Excel Models

In order to get a living breathing model, we decided to build it out in Excel. We produced a brief that required us to visualise the message queues and core applications as tabs in the spreadsheet and our use case actors (trade messages) as individual rows on these sheets. Our model would initially use additional tabs to hold the routing rules and reference data required to support the processing of our trade messages by VBA code. Our messages would move from tab to tab mirroring the processing flow in our use cases.

This proved to be a fast process and was handled solely within the BA team – without the need to disturb the developers. Whilst building this, we were forced to revisit our use cases several times in order to review the flow in various places and we could also see how our reference data requirements were maturing. To further enhance the model, we broke out the tabs containing our reference and routing rule data to MS Access and changed the Excel model to use that for its data requirements.

All this effort gave us the ability to run our use cases through the model (simple trade; simple trade with reversal and rebook; simple trade with give up etc). We could clearly see where we had issues around how we accessed our reference data and processed it. It was a superb model and it allowed us to walk through scenarios with the business and to bring the development team up to speed with what the Target Object Model actually looked like, how it processed, where data bolted in and what sort of things they needed to think about when designing their Adapter Framework.

There was only one thing missing from the model. This was the ability to asynchronously process all messages on all queues at the same time – just as our proposed Target Object Model would require. What problems would we encounter when we started pushing our use cases through the system? Could we really afford to wait until the integration testing phase until we discovered these problems? We didn’t know, and couldn’t wait, so we moved to the Cloud.

To the Cloud – Asynchronous Trade Flow Model in PHP/MySQL

We built out a Cloud-based version of our model, making use of all the information and knowledge we had acquired to date in the production of our play list, use cases and Excel models. For the technically minded amongst you, it was built using a PHP Framework and made use of a simple MySQL database backend. It was a relatively fast process to go through and we had something up and running within a couple of weeks.

We were able to drop messages (actors) into the Cloud model, see our Adapter Framework engage with those use cases and watch them flow through the message queues and into the core applications in our Target Object Model.

The model quickly highlighted the race conditions that we would encounter in our proposed system. It showed that our use cases needed to be expanded to include a “trade state” use case that applied business rules to the order in which status updates should be applied in the Target Object Model. It allowed us to see the issues around breaks and how resubmission would impact on the message queues. Finally, it provided a convenient hook for our test team to hang their test cases off, and prompted them to develop automated tools that would support integration testing, UAT and issue management.

Summary

All these steps gave the project knowledge:

  • Requirements capture and use cases were correct
  • The reference data model was mature
  • Asynchronous processing would not go into “melt down” at a business rule level
  • Appropriate test tools were built and made available for integration testing and UAT
  • The project had a tangible model of the Target Object Model for future projects in that domain

Not a bad result for a BA team to deliver to its colleagues! (Want more?)

Posted in Data Flow, Domain Model, Models, Trade Flow | Tagged , , , , , , , , , , , , , , , , , | Leave a comment

10 Things You’ll Do On Every Messaging Project – Designing an Adapter Framework

Posted on May 13, 2012 by Ben

This is the second in our series on life on an enteprise messaging project. This time we will be discussing that adapter framework you noticed appearing in the development team’s conversations. There’s no line for it on the project plan, and the client is *definitely* not going to want to pay for it.

DISCLAIMER Nothing in this article should be taken as a inducement or encouragement to engage in enterprise architecture unless supervised by a fully-qualified adult. The only thing more expensive than a good enterprise architect is a bad one.

What are adapters?

Shows System A feeding a message queue feeding System B

Adapters are the integration components that connect your messaging software to your core systems. You will have two basic types:
- ‘Outbound’ adapters: These poll your core systems for changes, and convert these changes into messages which they push to the messaging software. E.g. you might watch a SQL table and convert new records into a message
- ‘Inbound’ adapters: These poll a message queue for new messages, and convert these messages into API calls to a core system (assuming you’re lucky enough to have an API – the default one is called ‘stored procedure’)

What are adapter frameworks?

Adapter frameworks turn up as soon as you realise you are going to be creating quite a few adapters in your architecture. There are lots of things that ALL of your adapters will be doing, so it makes sense to solve these problems once, and then share these common solutions across all adapters. The rest of this article is about how long that list can get.

Logging and Error Handling

You can never have enough logging in a software system that doens’t have a GUI. You want to make it *really* easy for adapter developers to send frequent and verbose messages to the logging layer. You want log messages to include things like time, date and machine name automatically. If you allow multiple instances of the same adapter, you also need to correlate log messages to a particular instance. You might also consider rolling your log messages up to a single location, then you can get a unified view.

Error handling starts with logging – but you then should think about some general error handling strategies that you wish adapter developers to adopt. You should have a common set of negative unit tests that all adapters should pass:
- queueing system is down
- core system is down
- adapter host is out of disk space / memory
- poison message

Screen capture of Null Pointer Exception

You should also consider sanitising any messages that either support staff or end users will encounter. This could be addressed by a Development Standards Document as much as a code library.

Architecture

Unless you have a particular reason to allow cross-platform adapter development, you will want adapter code as either Java libraries or .NET assemblies, which in turn implies development languages, run-times and operating systems. Both are safe choices – any core system you encounter will probably have:
(a) A binary SDK for both platforms -or-
(b) A platform-netural API that both platforms support (e.g. SOAP, ODBC)

There are many standards that are relevant to the messaging world:
- AQMP for messaging
- JMX or SNMP to allow your adapters to report status
- WCF as an adapter accelerator if you are using .NET
- Common Logging
- Quartz for adapter poll schedules

As ever, it is more important that you have standards than the particular standard you select.

You should consider at what level different adapter instances share resources. Any combination of the following strategies are possible
- multi-threading inside a single adapter instance
- multiple adapter instances inside a single container (an example of a container would be Tomcat in the Java world)
- multiple containers inside a single run-time (either the .NET run-time or the Java run-time)
- multiple run-times on the same machine

These choices have an impact on manageability and thus your support department. A lot of support and monitoring software is targeted at the machine level, so you might think you are saving money on virtual machine instances by running multiple adapters on single nodes, but you are certainly going to be spending some money replicating or modifying systems like Hyperic.

Adapter Management

On larger projects, you will find that the challenge of configuring, deploying, versioning, controlling and monitoring adapters looms large. It makes sense to centralise as many of these functions as possible. Your adapters will need to be self-describing, and will ideally be discoverable to management systems, rather than needing to be explicitly registered.

There will be some configuration differences between an adapter polling a queue and an adapter pushing records into an SQL database, but these differences should be a matter of different fields being present, rather than requiring radically different administration procedures. Thanks to the  Development Standards Document, configuration settings will always include both a description and examples of valid values, so that it is easy for support teams to pick up what is going on with a new piece of software.

Adapter Business Logic

While your development team are building your adapter framework, your business analysts will be busy analysing and defining the actual messages that will shortly be winging their way through your adapters. Most analysts will present these specifications in either Excel or Visio (if you are lucky), but the foward-thinking architect may wish to consider embedding an ETL design tool such as Volante into the adapter framework. That way, specifications can be created directly in the GUI, and can be exported back into documentation at any point when paperwork is unavoidable. You may not be able to persuade your analysts to touch these tools, but they will certainly be a godsend to the development team.

You will also find that some message types are taken in and out of adapters in multiple places, particularly if you are thinking ahead and using a standard such as FIX. So you should also consider sharing your ‘message library’ across adapters. That way, updates can be applied consistently, or mistakes can be fixed in just one place (depending on how jaded you’re getting by now.)

Conclusion

With all this work going into the adapter framework, you’d be forgiven for wondering if this isn’t where most of the actual effort goes, and you’d be right. You’re actually creating an application template that may get used 10 or a 100 times through your project. But if you get it right, you’ll be creating and deploying new adapter instances in matter of weeks.

Posted in Architecture, Business Rules, Data Mapping, Domain Model, Enterprise Integration, Message Queue Software, Messaging | Tagged , , , , , | 2 Comments

Successfully Managing Business Analysts

Posted on May 3, 2012 by Graham

I am very passionate about the work that the Business Analyst (BA) does – I should be, I’ve been one for long enough. The role of the BA is critical to the success of any project. If the delivered system code keeps falling over, it can be fixed. If the system infrastructure is creaking, you can always throw another server at the farm. However, if you don’t capture the business requirements correctly, or if your BA team is inexperienced, or even if you just misunderstand them, you get complicated and costly problems.

I thought I’d walk through some of the pitfalls that can befall a BA work stream, having recently discussed a scenario I have seen before with an industry colleague. They work on a high profile, financial services project that is losing its way. The project appears to have been running for a considerable length of time and still appears to be at the requirements capture stage, with little in terms of artefacts and deliverables to show for it.

Stay within the project scope

BAs are incredibly inquisitive creatures. A good BA will uncover a thread, start pulling at it, keep pulling at it, trace it, follow it, absorb any information that thread unearths, consider and gain understanding of that information and will continue to doggedly do this until a satisfactory conclusion has been drawn. This is a wonderful approach to BA work and if your team are doing this then you have a winning set of BAs – a star team.

The only problem with this approach is that the BA’s understanding of a satisfactory conclusion does not necessarily meet the criteria laid down in your project scope. Working on out-of-scope items is a real killer for projects. Taking our unguided BAs, the work they do could very quickly become a week or two’s effort outside your project scope before the PM is aware and understands the issue. If you’ve got five BAs working on the project, then that two weeks easily becomes three months of combined lost effort, which few projects can afford to lose.

Work closely with the business

On site, I expect to see empty seats at BA’s desks, with the team out talking to the business, eliciting requirements, chasing down information, walking through use cases, reviewing wireframes and generally becoming an ever-present force in and around the business. There are times when this intelligence has to be shaped, ordered, tweaked and presented. However, even when that has been done, the business still needs to be walked through these presentations (unless you are lucky enough that your client actually reads your emails). Empty seats are thus a good sign of a healthy BA Stream – embrace it – don’t be afraid of it. If you see your BA team at desks all the time, then it’s time to question what is happening.

Share the vision – gaining business buy-In

This is a problem that arises when the BA stream begin to engage with the business. The business people you need to talk to are too busy to take time out to speak to you. This isn’t always the case as the business community have a vested interest in the delivery of a project/system that will make their life easier. However, this can be an indication of a communication problem from the stakeholders, where the vision has not been explained sufficiently to the business teams. This problem is common after the requirements capture phases of the project but it isn’t always limited to just that phase.

To tackle this and drive the project forward, a lot of diplomacy is required and you need to be able to manage the business community effectively. Sharing the project vision is often useful as staff in middle and back office roles are often overlooked by senior managers. Arrange convenient times to share this vision and try to get the team on side that way.

If this fails however, the problem must be resolved quickly lest weeks go by without BAs being able to secure sufficient time with the business. Everything is impacted, not just the BAs, and adopting the “can’t you be getting on with something else” approach to keep your team “busy” will often just exasperate the BA and compound the problem. Engage with your project and programme managers, talk to the project stakeholders and use soft power such as vision sharing to gain buy-in from the wider business community.

A roadmap for the BA delivery process

Those who have worked in Extraction, Transformation and Load (ETL) projects in the past will appreciate the distinct steps of:

  • E          Extract the data
  • T          Translate, make sense of, understand, format that data
  • L          Load that data to the required place

This is a simple roadmap that BAs should follow in their project activities. Development teams, testing teams and business colleagues need to be fed with information in order to work. For a BA, ETL becomes:

  • E          Capture/Elicit the requirements/problems/issues from the business
  • T          Assimilate the information, preparing it and formatting it accordingly
  • L          Present this information in a timely fashion to their consumers

A pitfall for BA work streams is to fall into the “analysis paralysis” trap. Through a lack of clear direction, the team loops endlessly through the E & T of the roadmap. This may be great for the BAs, satisfying their inquisitive nature, giving them an understanding of things both in and out of your scoping document. However, it is of no use at all to the project as a whole. The rest of the process requires clear, concise and accurate documentation in order to deliver the project. If your BAs don’t have a clear direction on what is to be produced and when, they may lose their way and spend months working very hard, only to produce very little. A Requirements Matrix ( http://ow.ly/aFOsC ) is critical to prioritising, executing and monitoring the BA work stream’s activities. Fail to create and maintain it at your peril!

Timely and effective delivery of BA documentation

The nature of BA work is that it involves the elicitation and assimilation of large amounts of information, and understanding these requirements is critical to the success of any project. Information gathered can be used to foresee potential issues in the Target Object Model and also to identify those areas of the “As Is” state ( http://ow.ly/aFIvq ) – problems perhaps even the current business/operations departments have not fully recognised.

But what use is it to the project to elicit that information, make that analysis, identify those issues and problems in the BA’s mind, if no documentation; no deliverable is produced?

For a project to succeed, the BA stream must document, document, document. The intelligence and information elicited needs to be stored somewhere where it can be seen and presented in a format that allows it to be easily consumed by the various people that need it. These include:

  • Your business colleagues who need the confidence that you understand what they need
  • Your development team who need to translate the requirements into code
  • Your test team who need to understand how to test the business cases
  • Your project manager who needs to know that risks are mitigated
  • Your programme manager who wants to be assured that costs are controlled

Retention of project and requirements knowledge

It goes without saying that BAs are needed in the early phases of the project. Their role is critical to the supply of accurate information to the development and test teams further into the development lifecycle. However, there is a strong desire to release them once this phase of the project has been completed. This is a waste of hard earned and costly information.

Often when the development team are working from the functional specifications, issues and problems arise that need the BA to iron out. If the BA has been released, it is then very difficult to understand the reasons for the decisions made by the BA. Retaining an element of the BA team is crucial during the build phases.

The BA and testing teams should also be very closely aligned. Bringing testers in early on the project and partnering them with designated BAs ensures a smooth transition into integration testing and UAT.

So don’t be too keen to lose your BAs. They have an accumulated knowledge and an intimacy with the Target Object Model that is not necessarily present in your development and testing teams. I’ve taken those post-contract calls from harassed programme managers to come back to projects and sometimes I have been available. But not always…

 

Posted in Project Management | Tagged , , , , , , , , , , , , , , | Leave a comment

Don’t dis the “As Is”!

Posted on April 16, 2012 by Graham

So, your project has kicked off. It’s big. It’s strategic. A career maker. Your task? To design and implement a new strategic messaging platform to replace the existing legacy system. This constitutes a myriad of SQL scripts, staging tables, tactical data stores and trade flows feeding several Core Applications. How hard can it be? Just rip the scripts and tables out and replace them right? A bit of data mapping and the job’s done yea?

Well, before you get carried away with the moment, take pause first. The following may well make the difference between a project that’s constantly playing catch up and one that delivers that Strategic platform, positions your firm ready for the future and makes your career as well.

It’s all too easy to see the end product in projects like this. To rush quickly into build, agile stand-ups and backlogs, iterative code releases and test driven development sprints. However, before you do any of this, you need to understand the here and now – the “As Is”. Without understanding that in detail, you cannot hope to succeed in replacing it.

How do you go about achieving this? How can you be sure you understand exactly what is happening within the current trade flow? There are a number of tools and techniques that you can employ to get to the bottom of the functionality. I’ll run through some of them here.

Tell me all about it – get the Business guys to talk about what it does and how it does it. Not just the 80% of the stuff that it does, get into the nitty gritty of the complex and strange stuff. If the system uses allocation rules – are all trades expected to match against them? You’d be surprised how many times you’ll hear an explicit “yes” answer, only to discover later on that a significant subset of trades never match against them and are just ignored by the current processes as later flow events superceed them. That may be fine for the “As Is”, but can your pristine messaging trade flow system cater for that without generating thousands of breaks?

The IT guys – without the involvement and buy-in of the current IT support team, it’s going to be incredibly difficult to understand the “As Is”. It’s not too bad if you are that current team, but it’s a challenge if you’re coming in from outside. Meet with them regularly, get them onboard. Try to have someone from that team assigned to you, either directly or via the matrix.

You’re so “As Was” – If your project is in flight for any significant length of time, it’s more than likely that the current system will not stand still whilst you work. The business will still want to drive forward, taking on new client requests; bringing new products to market; offering more visibility through in depth client reporting – it will never stop. Make sure there is a process in place that feeds changes, business developments and new project initiatives into your project and “As Is” model.

What goes in must come out – Do you know all the input and outputs involved in the current system? You may think you do but you need to consider not just the exterior boundary of the domain. There will be numerous internal boundaries between messaging and Core Applications as well as static and reference data that is required to support the processing. List them; categorise them; get samples and understand what the data means and how its meaning may change during the flow.

Timing is everything – you need to understand the sequencing of events in the current system in great detail. Which scripts run? When do they run? What data is available when they run? What data changes after they run? A mis-understanding here will cost you dearly further down the line.

If this; then do that – typically the business rules in the current system will have evolved over a number of years. They will have been scattered far and wide across the plethora of scripts employed in the system. You need to find them and understand them and to do this you will need to revisit the work you did with the business guys regularly. At this point you’ll start to unearth business rules that have been long forgotten by the current business community – these rules will have been implemented years ago by people who no longer work for the firm.

Once you’ve fought your way through the above processes, you’ll have enough information to produce the most valuable asset you will ever have – a working model of the “As Is”. If you produce this, you will reap the benefits time and time again. It will support your design process for the new system – which should produce it’s own model of the “To Be” domain model. It will allow you to validate your data mapping analysis steps. It will support your system testing steps and will also provide invaluable assistance it UAT/Parallel testing and issue resolution. It will allow you to compare “As Is” vs “To Be” states in discussions with the business, especially when the 20% edge case rule rears its head. In short, you’ll use this project artifact throughout the entire project lifecycle and it will reinforce your sanity time and time again.

So, if your BA’s aren’t talking about “As Is” and “To Be” domain models; if they are focussed exclusively on data mapping exercises; if they can’t describe the business use cases in the system inputs and they aren’t asking those probing questions around the edge cases and sequencing, all the things that feed into the production of the “As Is” domain model – then you need to take a long hard look at the project. Ask yourself the question – how can we possibly get to where we want to get to if we don’t know where we’re setting off from?

Embrace the here and now and Don’t dis the “As Is”.

Posted in Business Rules, Data Mapping, Domain Model, Trade Flow | Tagged , , , , , , , , , , , | Leave a comment

Ten Things You’ll Do On Every Messaging Project – Intro

Posted on March 30, 2012 by Ben

So you’ve read Enterprise Patterns, and understood at least half of it. You’ve either finally got Eclipse working, or downloaded last week’s service pack for Visual Studio. You’re ready to write your first enterprise messaging solution – but what’s it really like?

Here’s our hitchhiker’s guide to some of the realities of enterprise messaging:

1. You will spend a month designing an adapter framework

2. You will wish you had put more logging in your code

3. You will wish you could read messages on the wire

4. You will create an Excel document that shows how a single data field is mapped between seven systems

5. You will say ‘how hard can it be to write a messaging system?’

6. You will curse the ancient 1970s system that you have to integrate with, that the correlation pattern could have been named after.

7. You will have to add a field to your signed-off-in-blood message that was supposed to include every conceivable option

8. You will be asked to transfer Field A in System A to Field B in System B via your pristine common message format. No-one will be able to explain what this field is, but at the same time, it will be vitally important.

9. You will attempt to draw your system in PowerPoint, and switch to Visio because there’s not enough space on the page.

10. You will address a requirements change by renaming a queue or spinning up a new adapter, remember why you did this in the first place, and forget all the misery it took getting here :)

We’ll fill out each of these topics in more detail over the coming weeks.

Posted in Architecture, Enterprise Integration, Messaging | Tagged , , , | Leave a comment

AWS for Financial Services London – Using Big Data Analytic to Drive New Revenue

Posted on March 21, 2012 by Ben

Amazon are running a morning conference in the City next week (29 March 2012). We’ll be going along to find out more!

Posted in Cloud, Conferences | Tagged , , | Leave a comment