Success with Matching Engines – what does that look like?

 

match_engine

Implementing a Matching Engine application presents a host of challenges. If you’re responsible for such a project then you need to give serious consideration to a number of critical system components. Here are just a few of the questions that you’ll ne to think seriously about:

  • Is this a batch “slice and dice” implementation or more targeted and event driven?
  • Is the scope of your match data items clearly understood?
  • Have you quantified the range of operators that your match rules will need to cater for?
  • How will you test your matching rules and avoid overlapping result sets?
  • How will you manage your rules through quality assurance and release processes?
  • How will you know what to exclude from your results set in order to avoid false positives?
  • How will you see the audit of the match rule decisions?

The above questions are fundamental to your chances of success. You need to push your development team to bake the answers to these questions into your development from the start, otherwise you could experience legacy design problems even before you’ve gone live.

It is sometimes easy to mis-calculate the complexity of matching and rule engine implementations. Something that I try to keep in mind when tackling complex applications like these is a simple description of the problem space. For a matching engine implementation, I like to think of my problem as follows:

“I am trying to design, develop, test, implement and support a system that tries to compare a constantly changing dataset to another constantly changing dataset. I am trying to do this comparison with a constantly developing and changing set of rules. If my rules are not correctly configured and tested, I run the risk of polluting numerous downstream applications with the poor results from my application. I would desperately like to avoid being the cause of an implementation freeze in my organisation.”

So what can you do to manage your way through to a successful conclusion? Keep the following in mind during the project:

  • Deliver often and keep the business close during the development
  • Factor in time for rework – these projects often reveal their own requirements as they go
  • Build regression test packs and maintain them
  • Have your developers implement automated testing functionality to run these regression packs
  • Make sure that your rules are treated like any other project artifact – test them before releasing them
  • Have your developers tackle the problem of identifying overlapping matching rules

All the above will put you in a position of control. You know what sort of shape your application is in and whether the latest changes have introduced problems in functionality that was previously working. You can have confidence that you have rules management and testing controls in place, guaranteeing the quality of your releases.

You never know…

You may make such a success of it that you’ll be asked to do another one.

Posted in Business Rules, Messaging, Models, Project Management, Regression, Risk Management, Routing Rules, Rule Engines, Test Automation, Testing | Tagged , , , , , , , , , , , , , , , , , , , , | Comments Off on Success with Matching Engines – what does that look like?

Improving operational efficiency – Kanban and Basecamp

banner

We’re big fans of Basecamp here at Red Hound. We use it to collaborate on projects, to run our management activities, drive our recruitment and even for booking and approving leave. I wanted to share with you our recent experiences with trying to improve throughput and efficiency for a client team by using Basecamp as a platform for a Kanban board.

For those of you who may not have come across Kanban, this is a good place to start.

Our client’s team faced the following problems:

  • lots of tasks “on the go” with some items being “lost” in the system
  • “lost” items subsequently turning up as expedited work
  • low delivery rates from the team
  • no clear idea of the impact of unplanned tasks on scheduled work

Having read up about Kanban we thought it was worth a try. We’re not Kanban experts by any stretch of the imagination. However, the elements that interested us revolved around:

  • Maintaining outstanding, in flight and completed lists
  • Agreeing priorities at the start of the week
  • Limiting the number of in flight tasks

We started by creating a Basecamp project for the team and inviting all the relevant people to that project. We then agreed a visual emoji framework to indicate the relevant information for each task. Here’s our key:

emoji_key
We created a To Do list called Outstanding Tasks and added an entry for every currently outstanding task that the team was responsible for.

outstanding_tasks

We then agreed how many tasks we would allow to be active at any one time. Given that the team was three in size, we agreed that three would be a good idea to start with. We created a To Do list for those “in flight” tasks. Finally we created a To Do list where we would move tasks once they were completed. The Basecamp project looked a lot like this.

three_to_do_lists

We were then ready to go. The work flow that we agreed upon was as follows:

  • Agree the priorities for the week
  • Agree the tasks to be tackled this week
  • Drag these to the In Flight To Do lists
  • Work on the tasks
  • When completed, drag the tasks to the Completed list and pick up an Outstanding task

in_flight

In the normal operation of the Kanban board, the only way for a new task to be started is to complete an in flight task. Unfortunately, issues can arise in an unplanned fashion and sometimes tasks crop up that need to be worked on when the In Flight list is full. The beauty of the Kanban approach is that it is clearly visible when this situation arises. The discipline required is to “park” a currently In Flight task, move it back to the Outstanding list and then allow the unplanned tasks on to the In Flight list. The impact of switching tasks can be immediately seen.

So how did this help with the original problem of too many tasks in flight at the same time? The feedback from Emma, social media team member at The Write Angle: “Basecamp helped us to improve the efficiency of the team. The use of the Kanban board made it easier for us to see what we were working on and which stage it was up to. We also like the idea of “parking” tasks before allowing new tasks to be started.”

For us, it was a great success and seeing the improvement on the team was great. The collaborative nature of Basecamp provided the perfect platform for controlling, monitoring and allowing visibility on what the team were doing. There are other Kanban orientated software solutions available. However, for us, the flexibility of Basecamp allowed us to implement a lightweight Kanban process and Emma and her team reaped the benefits immediately.

Basecamp and Kanban – a great combination.

 

Posted in Business Rules, Kanban, Project Management | Tagged , , , | 1 Comment

Determining your AWS IP address range for a dynamic IP

To work with your AWS RDS PostgreSQL instance, you will need to provide your IP address or range. If you’ve got a dynamic IP address, this is the process to follow:

1. Determine your current IP: http://www.whatsmyip.org/

2. Look up the IP address range assigned to your ISP: https://apps.db.ripe.net/search/query.html

3. Convert that range into CIDR notation: http://www.ipaddressguide.com/cidr

4. Enter that range (or ranges) into your EC2 Security Group.

With thanks to Andy

Posted in AWS, Cloud | Tagged , , , , , | Comments Off on Determining your AWS IP address range for a dynamic IP

RabbitMQ – API Statistics – Documentation Trail

I appreciate I may be the only person in the world who cares about this, but every time I try and monitor a RabbitMQ server, I spend time digging through the documentation reminding myself what exactly I have to do. So here goes:

Step 1: Enable the RabbitMQ Management Console

Management Plugin
https://www.rabbitmq.com/management.html

Step 2: Download the CLI to inspect stats in a terminal environment

Management Command Line Tool
https://www.rabbitmq.com/management-cli.html

Step 3: Remind myself of how the HTTP API works

RabbitMQ Management HTTP API
https://raw.githack.com/rabbitmq/rabbitmq-management/rabbitmq_v3_5_0/priv/www/api/index.html

Step 4: Find the secret HTTP stats page (the stats link doesn’t work on the above page)

RabbitMQ Management HTTP Stats
https://raw.githack.com/rabbitmq/rabbitmq-management/rabbitmq_v3_5_0/priv/www/doc/stats.html

That’s it – apart from figuring out how to parse JSON with grep, but that’s another story.

Posted in Message Queue Software, Messaging | Tagged , | Comments Off on RabbitMQ – API Statistics – Documentation Trail

A Problem Shared

Onwards and Upwards

Do you work in a highly fluid IT environment? Do you work in or manage a geographically disparate virtual team? Do you struggle with building in and maintaining the quality of your developments and releases? Then this article could be for you, we take a look at Red Hound Consultancies’ experience of operating a “Pair Programming” approach to successfully address these issues…

A common approach for many consultancies is to allow minimum team size of a single person often working part time.  This has the following disadvantages:

  • Poor requirements or technology documentation can have a big impact on a single person. Trying to cope with chaos is much easier when shared
  • The sense of isolation can unnecessarily trigger off panic
  • Development, Test and Documentation tend to run in traditional waterfall sequences
  • Quality can often be poor for all the above reasons
  • Any strengths are typically down to the individual
  • Individuals can “disappear” when working virtually either intentionally or through despair!
  • Single points of failure are prevalent i.e. individuals on leave of if they actually leave or if unforeseen circumstances occur – This is high risk for company and client alike!

Red Hound’s experience has shown that with the simple expedient of constructing your teams with a minimum size of 2 people the above blockers are instantly removed and what’s more this doesn’t cost you or your client anymore and doesn’t constrain your resources either. (This concurs with the experience of others – see Recommended Reading)

Specifically Red Hound recommends you consider the following:

  • Maintain your initial client estimate for man days (don’t make it cost more)
  • Allocate at least 2 people – they don’t have to be full time and you can mix and match e.g. 1 full time and 1 part time or any combination there of
  • In conjunction with or based upon your projected resource allocation, decide if you maintain the original delivery date or bring it in
  • Within the team, structure the work so that where possible activities run in parallel e.g. Dev and Test of multiple iterations. Importantly QA can run in parallel – we have found observing code development in real time significantly reduces errors

A small word of caution, you need to ensure that your pairs work successfully together and keep an eye out for issues developing such as personality clashes or the Expert dominating the Novice etc; these can be addressed either at the recruitment stage –hire the right person and set expectations from the start – or via management review.

We have been operating in the above fashion for one of Red Hound’s high profile customers and have realised all of the predicted benefits. The client has given Red Hound a very high satisfaction rating!

In summary, for no extra cost to you or your client simply creating minimum team sizes of two people drives quality and productivity across the board whilst simultaneously boosting team morale and eliminating single points of failure. Give it a go – you have nothing to lose and a problem shared…

Recommended Reading

http://en.wikipedia.org/wiki/Pair_programming

http://blog.codinghorror.com/pair-programming-vs-code-reviews/

http://www.uxbooth.com/articles/write-better-content-by-working-in-pairs/

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.101.9212&rep=rep1&type=pdf

Posted in Agile, Project Management | Tagged , , , | Comments Off on A Problem Shared

ICE Collateral Balances Report – two digit years and Y2K workarounds

What year is it?

That pesky Y2K issue is back to haunt us …

ICE are providing maturity dates in the Collateral Balances report with just two digits for the year. These maturity dates can be thirty or more years into the future. And that takes us into the territory of the Y2K workarounds for two digit years.

The default behaviour for most databases is to use a cutoff to interpret two digit year values in dates. The cutoff for SQLServer is set at 2049 so any year entered as “00” thru “49” is taken to be 2000 thru 2049. Any year entered as “50” thru “99” is taken to be 1950 thru 1999. Oracle follows a similar pattern based on a cutoff of 2050.

The value of the cutoff is adjustable in the database configuration options. If your system only deals with 21st century dates you could set the cutoff as 2099 so that all dates will be given “20” as the century.

But sadly that is not the end of the story …

Where data is being loaded into a database through an external toolset then it is the toolset that controls interpretation of dates. Windows 7 OLE Automation is set with a cutoff of 2030. Hence a two digit year of “31” thru “99” will be stored into the database as 1931 thru 1999. Microsoft do not provide a means to alter the OLE Automation cutoff.

JDBC gives a cutoff at 2034 and again there is no simple way to adjust that cutoff.

So it really is time to persuade your data providers to supply four digit year values on all reports.

Just as they should have been doing since Y2K first threatened to end the world!

Posted in Data Mapping, Derivatives Industry, Regulatory Reporting | Tagged , , , | Comments Off on ICE Collateral Balances Report – two digit years and Y2K workarounds

Enterprise Integration Projects – How do I know when I’m done?

Being responsible for a large enterprise integration project brings with it an ominous set of challenges. If you’re project managing such an endeavour then you need to have a constant mind on how you can prove that your integration pieces are working. The problems are complex and you need to be asking yourself the following:

  • How do I prove that my data mapping has been done correctly?
  • How do I prove that my message sequencing is working?
  • How do I prove that my break management is working?

This is a very challenging set of questions. One thing you will find is that if you don’t force your development team to think about these in advance, then you’re going to struggle to answer these questions effectively.

To get a true feel for the complexity involved, think about your burden of proof challenge like this:

My burden of proof challenge = A x B x C x D

  • A = number of messages
  • B = number of different message conversation types
  • C = complexity of each message
  • D = number of places each message needs to go

If you have a million messages per day, eight different conversation types, a complexity factor#1 of 5 and 8 down stream adapters/systems that you need to publish to you’ve got 320 million spinning plates that can fall at any time.

A good proportion of your testing (and to some degree your development) effort should be directed to the tasks that surround each of the above elements. Ask yourself, if you have 230k breaks in a million trade messages:

  • Can I actually quantify the problem?
  • Are there really 230k individual issues?
  • Can I categorise the issues appropriately?
  • Can I automate this analysis and repeat it?
  • Can I give my process a “memory”?
  • Can my process recognise the same patterns arising again?

Applying the correct resources and knowledge to each of these questions will allow you to move closer to answering the most critical question you need to answer on any system integration project:

How do I know when I’m done?

Red Hound have an excellent track record of providing definitive answers to what can sometimes seem almost rhetorical questions. Read more of our blogs for insights in to how we have brought focus to diffuse requirements and exceeded customer expectations. Alternatively, why not contact us to see how we can help you….


#1 – I calculate the complexity factor as follows:
Complexity Factor = 1 + Integer Value of (number of fields on the message divided by 10)

Posted in Data Flow, Data Mapping, Enterprise Integration, Messaging, Project Management, Trade Flow, Uncategorized | Tagged , , , , , , , , , , , , , | Comments Off on Enterprise Integration Projects – How do I know when I’m done?

Not modelling your workflow? Here there be monsters!

I wanted to share a recent project experience with you that further strengthened my belief that a picture paints a thousand words. It helped to identify the root cause of a show-stopping problem where other efforts to do so had failed. ().

The project landscape:

  • A client-enforced, aggressive and fast approaching implementation deadline
  • A trade messaging project with complex and distinct business and routing rules silos
  • Messages from multiple asset classes sourced from several upstream trade capture systems
  • Performance issues requiring a hasty workflow redesign
  • Insufficient time to document the complex business/routing rules and workflow

The symptoms of the problem:

  • A persistent issue on a single asset class
  • An inability to reproduce the problem in the debug testing rig
  • A clash between the business and routing logic producing an “intermediate” status
  • Upon replaying the message the problem corrected itself

Steps taken to date:

The project team attempted to reproduce the problem locally in their testing rigs. It proved impossible to do so. In the testing rig, the clash between business and routing rules did not occur. The trade messages were processed as expected.

A deep dive was taken into the business and routing rules. Where was the conflict? The business rules said “yes”. The routing rules said “no”.

It was not possible to reproduce the error. Without being able to reproduce it it was not going to be possible to find the root cause and resolve it.

The clock was ticking and this problem had been ongoing for four days already.

What Red Hound did:

When the Principal Consultant shared the problem with me my thoughts turned to the workflow model. I could use that to walk through the model and “be” a trade message. Unfortunately, there wasn’t a workflow model. It had never been produced – a step dropped in the pressure to deliver. Unfortunately, I knew this problem wasn’t going to be solved without it, so I took myself away from the maelstrom in order to draw it out.

After a couple of hours of documenting the flow, creating the model, walking trade messages through it and analysing the business and routing rule silos the issue started to reveal itself.

The redesign had introduced an anomaly into the model. It had resulted in adapters in the main flow that retained cross asset class entry business rules but single asset class exit routing rules.

There hadn’t been time to refactor the engines and rule sets.

main_workflow My workflow model clearly showed me that the message replay workflow had not been redesigned and retained its single adapter with cross asset class entry business and exit routing rules.replay_workflow

As the symptoms stated, replay worked but the main workflow failed.

Was it possible that somehow the failing asset class was being processed by the wrong adapter? The entry business rules in the main flow would allow any asset class into the adapter whereas the exit routing rules could only successfully process a specific, single asset class. Was it possible for the business rules to say “yes” but the routing rules to say “no” in this scenario? It was if the wrong asset class was being processed in an adapter.

The solution:

A check with the infrastructure team revealed that a sizeable minority of Rates trades were being mistakenly pushed down the FX queue into the FX adapter. The cross-asset entry business rules in the FX adapter processed the Rates trades successfully, resulting in a “yes” state. However, the FX exit routing rules didn’t recognise them and so dropped into their default mode resulting in a “no” state.

The queue routing was corrected and the problem went away, as my workflow model predicted.

Conclusions:

  • The starting point of all workflow projects must be a workflow model
  • Creating your workflow model is not a waste of time. You do have time to do it and the return on your investment will be worth the effort
  • Start workflow redesigns by updating your model. This will reduce your risk exposure and allow you to quickly identify potential flaws
  • Infrastructure changes must be documented and communicated – your workflow model is a great basis for this
  • Build smoke testing packs to give you confidence that infrastructure changes have been successful
  • Bake logging into your adapter framework
  • When problems arise in the system, use your toolkit- including your workflow model to help identify the root cause
  • As far as RULES and WORKFLOW are concerned, the code NEVER documents itself

By building out the workflow model and walking it, I was able to resolve the problem that had been plaguing the project for four days. A couple of hours effort well spent.

()

Redhound are happy to advise on all aspects of business analysis modelling, including reference data, business process and trade flow.

Get in touch!

We hope this post has been helpful.

If you’d like to find out more about our approach, the technology we use and the partners we work with, get in touch

Posted in Business Rules, Data Flow, Domain Model, Messaging, Models, Replay, Risk Management, Routing Rules, Rule Engines, Smoke Testing, Trade Flow, Uncategorized | Tagged , , , , , , , , , , , , , , , , , , , , , | Comments Off on Not modelling your workflow? Here there be monsters!

The next time I implement a time-sheeting system

Salvador-Dali-soft-clock-300x243I will not punish my consultants for being billable on client site by requiring them to log into a VPN, use IE6, use ActiveX downloads, or place other cruel and unusual barriers to them billing time.

I will not make my consultants learn Sage codes, SAP codes, Navision codes, Oracle Financials codes or any other accounting system that man has invented. My consultants should know the client, the project and the hours they have worked. My finance team can work out how clients and projects map to their systems.

As an exercise, I will design a Word template which captures everything I need, along with a space for the client to sign, which my consultants can print out, complete and post to my office. This is admittedly a stupid, slow and painful solution. Any technology solution that manages to be stupider, slower and more painful should be ruled out. Whatever technology solution I chose, I will make sure that everyone involved understands the fall back plan is the Word template. It is not putting our time recording and invoicing process on hold.

I will ask any consultant involved in the adoption process how hard they think it is to deliver a time-sheeting project. Anyone who thinks it is easy can try parsing a date string in Java, and come back when they’re done.

I will assume that public holidays, leave and sickness are certainties for any human consultant, and make it easy for them to log time against these events, however else I decide to control availability of projects and clients.

I will operate a one-in, one-out policy for time-sheeting systems. If it is cheaper for hundreds or thousands of consultants to enter the same data in two different systems than to integrate those systems, I will find a cheaper system.

I will be extremely wary of ‘time-sheeting modules’ for my accounts package, my ERP package, my project planning package or my content management system. It would admittedly be nice if one system did everything. But it would be better if my time-sheeting system was usable. And worked on client site.

I will remember that my managers will be keen to control consultants recording time against the ‘wrong’ activities. I will be open-minded about any technology measures that are put in place to support these controls. I will also ask my managers to operate the Word template system in the event of these controls preventing anyone recording time.

I will ask any supplier ‘how do I get time records out of your system.’ I will then ask ‘how do I get my clients, projects and staff into your system.’ I will carefully consider the time it takes to do these things against the number of time records, clients, projects and staff I have to deal with.

Posted in Architecture, Project Management | Tagged , , | Comments Off on The next time I implement a time-sheeting system

You are not doing Scrum

Biohazard sign

You are not doing SCRUM

Because your stand-up is a 45 minute conference call to India.

Because you still have to estimate total effort and duration.

Because the customer does not attend reviews or planning meetings.

Because you are not showing working software at the end of your sprint.

Because you are following an “Agile” process mandated by a central PMO team.

Because project priorities still change on a daily basis.

Because you don’t monitor burndown and feed back what you see into planning.

Because your team leader just assigned you a JIRA.

Because the definition of complete for the task you are working on is in your manager’s head.

Because you feel relieved when the daily stand-up is cancelled.

You are not doing Scrum. You are just spending more time on the phone.

 

Posted in Agile, Scrum | Tagged , , , | Comments Off on You are not doing Scrum