www.vExperienced.co.uk

This year my VMworld experience started in a more relaxed fashion than previously as I flew in ahead of time on the Sunday night. After checking in to my hotel and getting my orientation in the city I headed (along with LonVMUG’s Luke Munro) to the vRockstar party at the Hard Rock Cafe organised by Marco Broeken and Patrick Redknapp. This coincided nicely with ‘El Classico’ when the two giants of Spanish football, Real Madrid and Barcelona, play each other in the Spanish league. This ensured the Hard Rock Cafe was rammed full so it was a good thing they’d reserved an area for us. Food, (free) drink, and good conversation – thanks for organizing a great start to VMworld guys!

Next day registration at the conference venue was very quick partly because it was partner day and the masses had yet to arrive. There was some misleading information about the HOL being closed although after a quick Twitter shoutout to John Troyer that was quickly remedied. As I’m a customer not a partner I didn’t have access to the partner breakout sessions so I figured my day was going to be a mixture of labs and people networking. Compared to Copenhagen the weather was a distinct improvement, hovering around 25 degrees and quite humid, although inside the air conditioning kept everyone cool.

The Keynotes and announcements

Tuesday signaled the first day of the main conference when all 7000 attendees turned up. The day started with the keynote from Pat Gelsinger and Steve Herrod and was largely a repeat of the US keynote with a few notable exceptions which I’ll cover later. For those that haven’t seen the US keynotes here’s the highlights;

• there is a new vCloud Suite which bundles many of the VMware products together in a more compelling and cost effective package
• vRAM is no more (cost is now per socket)
• the launch of vSphere 5.1
• new certification tracks including a vCloud track

VMware always like to hold back some product launches so that VMworld Europe has something to get excited about. Here’s a summary of the announcements at Barcelona;

• Dynamic Ops has been integrated into the vCloud Suite and rebranded to vCloud Automation Center 5.1.
• Updates to vCloud Connector and vCenter Operations
• Updates to the vFabric line including Application Director and Application Performance Manager.

With the swift integration of the Dynamic Ops technology VMware obviously want to manage heterogeneous clouds having spent the last five years saying there was no demand. Should we take this as indirect endorsement of Hyper-V?

Read more…

If you’re in the market to take a VMware certification exam, there’s some good news – provided you’re quick. For the next couple of days (while VMworld Barcelona is running, Oct 9th-11th 2012) you can book the VCP and VCAP exams at a cool 50% off. For VCP that’s a saving of approx £50 and more like £200 for the VCAP exams! If you want to blitz some of the new certification tracks recently announced you’re not limited to just one – study your little legs off and you could save even more by taking multiple exams….

The codes you need to register with are;

• VMWBAR50 – for the VCP exams (VCP-DV, VCP-DT,VCP-Cloud etc)
• ADVBAR50 – for the VCAP exams (VCAP-DCA, VCAP-DCD etc)

Conditions:

• You MUST book the exam while VMworld Barcelona is running. You don’t have to be attending the conference, it’s just the period of time the offer is valid.
• You MUST take the exam by the end of the year.

What are you waiting for? Head over to VMware Certification and get registered certification junkies!

While working recently on an ADFS federation solution I came across a Microsoft ‘feature’ which doesn’t seem to be well known and which caused me to deliver my project a week late. It often manifests itself via failed logins and affects many products which integrate with AD such as Sharepoint, Office365, OWA, and of course ADFS. This is very much one of those ‘document it here for future reference’ posts but hopefully it’ll help spread the word and maybe save someone else the pain I felt!

To describe how the ‘feature’ affects ADFS you need to understand the communication flow when a federation request is processed. The diagram below (from an MSDN article on using ADFS in Identity solutions) shows a user (the web browser) connecting to a service (the ASP.NET application although it could be almost any app) which uses ADFS federation to determine access;

Communication flow using federated WebSSO

Summarising the steps;

• The user browses to the web application (step 1)
• The web app redirects the user to ADFS (step 2,3)
• ADFS attempts to authenticate the user, usually against Active Directory (step 4)
• ADFS generates a token (representing the users authentication) which is passed back to the user who then presents it to the app and is given access (steps 5,6,7)

My problem was that while some users were being logged into the web application OK, some were failing and I couldn’t work out why. Diagnosing issues in federation can be tricky as by its nature it often involves multiple parties/companies. The web application company were saying their application worked fine, both redirecting users and processing the returned tokens. The users were entering their credentials and being authenticated against our internal Active Directory. ADFS logs showed that tokens were being generated and sent to the web app. Hmm.

Digging deeper I found that the AD username (the UPN to be precise) being passed into the token generation process within ADFS was occasionally incorrect. The user would type their username into the web form (and be authenticated) but when ADFS tried to generate claims for this user via an LDAP lookup it used an incorrect UPN and hence failed. It seemed as if the Windows authentication process was returning incorrect values to ADFS. This stumped me for a while – how can something as simple and mature as AD authentication go wrong?

Of course it’s not going wrong, its working as designed. It transpires there’s an LSA cache on domain member servers. On occasions where the AD values have changed recently (the default is to cache for 7 days) it can result in the original, rather than the updated, values being returned to the calling application by the AD authentication process. A simple change such as someone getting married and having their AD account updated with their married name could therefore break any dependant applications. Details of this cache can be found in MS KB article 946358, along with the priceless statement “This behaviour may prevent the application from working correctly“. No kidding! This impacted my project more than most because the AD accounts are created programmatically via a web portal and updated later by some scripts. The high rate of change means they’re more susceptible to having old values cached.

This might seem like a niche problem but it also impacts implementations of Sharepoint, OWA, Project server, and Office365 – any product that relies on AD for authentication. These products can be integrated with AD to facilitate single sign on but if you make frequent changes to AD the issues above can occur.

How can I diagnose this issue?

The symptoms will vary between products but thankfully Microsoft have some great documentation on ADFS. The troubleshooting guide details how to enable the advanced ADFS logs via Event Viewer- when you’ve got those check for Event ID 139. The event details shows the actual contents of the authentication token so you can check the UPN and ensure it’s what you expect. If not follow the instructions in the KB article to disable or fine tune the cache retention period on the domain member server (ie the ADFS server, not the AD server).

Further Reading

Understanding the LSA lookup cache

Having had some time to digest at the announcements from VMworld 2012 (day one, day two) I was reminded of the childrens story about the hare and the tortoise. Yes it’s another ‘analogy post’ but otherwise technology can be so bland!

The story tells of a hare that can run so fast, no-one can beat him. The tortoise, slow moving by nature, challenges the hare to a race and the hare, laughing, accepts knowing he can beat the tortoise with ease. On the day of the race they line up. Bang! The starter’s pistol goes and off goes the hare, charging into the lead. After a while he looks back to see the tortoise miles behind. Seeing how much time he has he decides to take a quick sleep under a nearby tree. When he wakes however he realises that the tortoise has passed him by and he’s unable to catch up so loses the race. While not a perfect analogy I think VMware is the hare and their customers the slow moving tortoise (no, the tortoise is not Microsoft, how unkind…). VMware are creating technologies and an ecosystem at a speed which customers are struggling to adopt, and much of this week’s developments are because of this imbalance (or ‘virtual stall’ as Andi Mann coined it). Pat Gelsinger, the incoming CEO at VMware was quoted comparing the company to ‘an adolescent who has grown too quickly’ because their operational rigour hasn’t kept pace with the company’s growth. It’s not only customers who are grappling with the pace of change.

Let’s look at pricing. VMware have binned the consumption based vRAM licencing scheme and reverted to their per socket model used prior to vSphere5. This was an unpopular scheme and with Microsoft’s Hyper-V hot on VMware’s heels I think VMware realised that to stay competitive they had to react. While many applauded the u-turn it’s been pointed out that the future of cloud is all about charging for usage (here and here) so maybe VMware were just ahead of their target market? If the dynamic environments promised by IaaS were commonplace then maybe Microsoft would have been amending their licencing rather than VMware making an embarrassing (though brave) climbdown.

VMware still have the dominant virtualisation portfolio, certainly within the enterprise, but they need to leverage it to maintain their premium pricing and hence profitability. Products such as vCloud Director, vFabric, and vCOPs haven’t seen the uptake VMware were hoping for and without these ‘value add’ tiers the core virtualisation product isn’t remarkable enough to counter the threat from rivals like Microsoft and the open source community. People have been wondering when Hyper-V and Xen will be ‘good enough‘ for a couple of years and many think the time is now. VMware have the technology and the vision but many customers aren’t ready to implement it. We’re still talking about only 60% of server workloads being virtual, and getting tier 1 apps like Oracle virtual is taking a long time (due to FUD and Oracle’s desire to own the whole stack as well as technical factors). Automate my workflows? My company are still struggling to even define new manual workflows and processes given the huge changes that virtualisation brings to any large company. Move to the cloud? Half our production servers are still physical. VMware still have a strong market position but the longer customers take to move to the new technologies, the greater the opportunity for competitors. The hypervisor is already a commodity – if customers take many years to move to the next stage then the management stack that VMware are now pushing may also be a commodity.

VMware need their customers to accelerate their move to the cloud before their product line becomes a commodity. How are VMware tackling this?

• They’re integrating their products to ease implementation. The newly announced vCloud Suite is just bundling and minor compatibility tweaks but over time I imagine it’ll become a much more joined up offering. Server virtualisation was an easy first step whereas ‘the cloud’ requires more than just vCloud Director and that’s not well understood.
• They’re bundling vCD with Enterprise+ (free therefore to a good chunk of their customer base) and making it significantly cheaper for customers on lower licencing levels (especially now it’s per socket, not per VM).
• They’re offering easier vCloud ‘test drives’ via the partner channel
• They’re creating a ‘cloud forum’ to help customers adopt to the new way of thinking.
• They’re expanding their training and certification to specifically address vCloud (as well as more focus on the desktop and application stack). IT pros are struggling to keep up with the new developments too and without a skilled workforce companies can’t move forwards. This is also behind the ever expanding vExpert program and the growing number of VMware Evangelists.
• They’re preparing for a more heterogeneous world with their Dynamic Ops acquisition and their membership of OpenStack – if you can’t win the race outright at least make sure you’re competing for a place.

Is it out of VMware’s hands?

Obviously VMware can use the above actions to spur adoption to their cloud specifically (to speed up the tortoise in my analogy!) but mainly it’s market forces which will drive the change – ‘cloud’ is one the hottest areas and is set to grow. Speed may be the key – if the enterprise masses don’t migrate to the cloud for another 5-10 years there will be increased competition and VMware risk losing the premium value in their products and potentially their stellar profits. If we’re still talking about virtualising tier 1 apps, the year of VDI, and how to integrate a full cloud stack in another couple of years (which I suspect we will be) it’ll be interesting to see if VMware can maintain the place on their top of the podium. Despite cloud being considered mainstream I think there are many who remain tortoises, plodding along. And in the original Aesop’s fable it’s the ‘slow and steady’ which win…

Right Here, Right Now is the tagline for this year’s conference. To use another FatBoySlim title I’d say “Halfway between the gutter and the stars” is more appropriate.

A recent project at work has required me to implement Microsoft’s Active Directory Federation Services (ADFS) which has been an interesting change from my usual technologies. It’s a mature product (it was released with Windows 2003 and further refined in Windows 2008) designed to allow you to ‘federate’ your Active Directory – in other words to allow third parties to leverage your internal AD in a secure manner. At first I thought this project was a distraction from the skillset I’m working towards (IaaS infrastructure with vCloud Director, View etc) but I’ve since come to realize that federated identity is an essential ingredient in the cloud recipe and one which needs to be understood.

Let me give you an example. My company decided to upgrade an aging training application and for various reasons we outsourced the solution to a third party developer. The idea was that they’d develop (and host) all the training materials and offer it as a service over the web to our customers (SaaS) thus requiring no resource from our internal teams. The only hitch in the plan was the business requirement that the customer, who already has login details for our web portal, should use the same credentials for this remotely hosted training solution. Those credentials are held in an internal AD database so we used ADFS to ‘publish’ them to the third party. Voila! The end users can now login to their training solution unaware that the credentials they enter are authenticated against my AD in the background. The resulting (much simplified!) architecture is shown in the following diagram;

It’s important to realise that there are two distinct actions going on during a login;

1. Authentication – The AD acts as the identify provider (IdP), making sure the user is who they say they are.
2. Authorisation – Once authenticated ADFS generates a ‘claim’ which it sends to the third party and this dictates what actions the user can take in the application.

Of course my example is very simplistic but the principle of allowing identities to be shared securely across security boundaries (such as disparate networks and applications) is critical to cloud services. Security is one of the big challenges in the cloud and federation allows you to keep your crown jewels (your user details) secure while still consuming remote services. It’s also important as the number of mobile devices used to access services increases.

Consumer or provider?

The ADFS example above is just one of the many possible scenarios that federated identity must handle. In every federation scenario there is an identity provider (IdP) and a consumer or service provider (SP, sometimes referred to as a relying party). In the the example above my company are the identity provider (our AD holds the identity details) and the consumer is the third party developer who provides a service.

The first choice therefore is whether you’re just going to consume other people’s federated identity services and/or act as an identity provider yourself.

You’ve probably been a consumer of federated identity for a while without even realising it. Everytime you sign into a website using your Twitter or Facebook login (for example) you’re consuming the federated identity service offered by Twitter and Facebook, likewise when you post a comment on a blog which requires a WordPress login. Maybe you’ve logged into a variety of Microsoft services using your Windows Live ID? Same thing.

One of the early commercial attempts at Federated Identity was Microsoft’s Passport which set out to be a universal authentication mechanism for web commerce but security concerns limited it’s adoption and resulted in a proliferation of alternative services (Windows Live ID, Google ID, and Apple ID to name a few well known ones). Here’s a few of the most popular federation protocols in use today;

• Open-ID (which was formed by Facebook, Google, IBM, Microsoft, PayPal, VeriSign and Yahoo)
• OAuth (similar to OpenID but used for API delegation, used by Twitter, Salesforce, Google, Facebook etc)
• SAML (the most widely used federation protocol used by ADFS, Horizon App Mgr, Centrix Workspace and others)
• WS-Federation (part of the larger WS-* standards)
• SCIM (the newest and still evolving standard – v1 was ratified in Dec 2011)

So consuming is commonplace but why would you want to become an identity provider and federate your identity out to the world?

You might be reading this article and thinking ‘I don’t offer a service to people on the internet so I’ve no need to provide identity federation’. If all your infrastructure needs are met internally that might be true. What if you want to use public or hybrid cloud? If you want your corporate users to securely use their company login to access SaaS providers like SalesForce.com or Google Apps you’ll need to become an identity provider.

If you’re an internet giant like Google, Microsoft, or Apple you can develop your own identity framework but for everyone else there are frameworks you can quickly ‘bolt on’ to your existing infrastructure which allow you to offer federated services;

• Microsoft’s ADFS (and the upcoming Windows Azure AD)
• OpenAM (open source)
• Shibboleth (open source)
• IBM’s Tivoli Federated Identity Manager (TIM)
• NetIQ Access Manager
• RSA’s Federated Identity Manager
• Site Minder (CA)
• Centrix Workspace
• VMware Horizon Application Manager

The purpose of the above applications varies even though they all provide identity federation. Most include SSO functionality but some are cloud based and others are installed locally (some are deployed via appliances). Some provide ‘application store’ or portal/workspace features which are much like the Citrix access you’re probably familiar with but for both internal and cloud applications.

I was already familiar with the Centrix solution after seeing one of the company founders, Lisa Hammond, give a very good presentation at the recent July 2012 London VMUG. The idea of a converged portal presenting SSO access to all your apps, wherever they reside, is compelling and Centrix has been doing this for quite a while prior to VMware’s entry into the market.

How is this relevant to me as a virtualisation admin?

You’ll have spotted the last entry above, VMware’s Horizon Application Manager. Horizon was released in May 2011 as the first component in the ‘Project Horizon’ vision first previewed at VMworld 2010. It was developed from VMware’s acquisition of TriCipher in August 2010, a company which previously developed a federated identity solution known as MyOneLogin. To quote VMware’s press release at the time;

VMware’s acquisition of TriCipher lets us integrate identity-based security and managed access to applications hosted in the cloud or on-premise. Convenient end-user access to applications on any device with security controls for IT lets customers extend their security and control into public cloud environments.

Earlier this year VMware published a research article on identity, access control, and Horizon which is a great introduction to Horizon and where it fits in the larger ecosystem. VMware would like you to implement Horizon to act as a centralised portal for all your applications whether they reside internally or externally via clouds. Another way to think of it is an ‘app store’ for the enterprise. Michael Letschin (@mletschin) has written a very clear roadmap for the encompassing Project Horizon vision which as a bonus also makes clear where Project Octopus fits in – a great read. For a practical understanding of federation and VDI check out Andre Leibovici’s great article discussing federation in relation to VMware’s View product although he advises that it’s still very much a work in progress.

The principles and terminology of federation (IdP, SP, tokens, relying parties, claim rules etc) are largely the same across all the products listed above so I’m glad that by learning ADFS I’ve actually learnt quite a bit about how Horizon works under the hood.

The bottomline is that if you’re going to use cloud services and you want to avoid a security management nightmare you need to understand federated identity. if you don’t understand your options early on you may find yourself putting in a solution which solves your short term requirements but not your long term goals, and that could lead to implementing multiple solutions – messy!!

This article barely touches the surface of a very complex subject – ADFS is fine if you’re using Microsoft as your on premises identity provider, but what if you use another user directory from Oracle or IBM? What about two factor authentication? What if a third party uses open source Shibboleth and you use ADFS – do they work together? Can you chain authentication systems together and introduce conditional processing? What about multi-tenant clouds and the special challenges they present? Federated authentication is one small part of a wider subject commonly referred to as Identity Management (IDM). I did enough to get our implementation working but it was immediately obvious that it’s a specialised skillset every bit as complex as virtualisation with multiple products, protocols, compatibilities, design choices and pitfalls. I also found it fascinating to see how the various disconnected services are beginning to be ‘hooked’ up to each other using these distributed mechanisms – there’s a long way to go but this is a growth area no doubt.

Further Reading

Here’s a great primer on using ADFS, explained in real world terms (although the concepts apply to any federation)

A good intro to ADFS, particularly when used with Office365

A good video intro to federated identity

Will OpenID transform the enterprise ecosystem?

Federated clouds – Possible?

SCIM: Standards are taking hold in the cloud

Good article on ADFS and the Azure version of AD

Looking at cloud futures – why federation is key to cloud evolution

The essential OAuth primer

Hosted application stores help IT pros wrangle cloud apps

Centrix gave a very good presentation at the July 2012 London VMUG

UPDATE: 21st August 2012 – CRN have leaked news that VMware’s controversial vRAM licencing may be on it’s way out. No doubt we’ll find out at VMworld SF next week.

I like conspiracy theories, and this is mine for today. Back in 2009 VMware introduced the Enterprise+ licencing tier to the frustration of existing customers who felt they were already paying top dollar (especially the large enterprises with site agreements). They simultaneously announced that the existing Enterprise tier would be discontinued and offered a 50% discount on the upgrade to Enterprise+ but after disappointing takeup the Enterprise licence was allowed to persist. As the vSphere product suite has evolved  over the last three years the bulk of the new features have gone into the Enterprise+ tier but I still speak to plenty of users who aren’t prepared to pay the extra. As competitors (Hyper-V, Xen etc) improve VMware need to differentiate themselves but that’s hard to do if a chunk of your customers haven’t adopted your latest, greatest, features.

That’s where the new vRAM licencing model introduced with vSphere5 has a part to play. While not the driving reason for the move to this usage based pricing model it does mean that as hardware scales up (we’ve just gone to 192GB per dual socket server) you might find yourself moving to Enterprise+ just to get the extra vRAM allowance. Recently I’ve had to determine the best way to cope with a significant increase in the number of VMs we host (about a 170% increase) and the most cost efffective option was to move to Enterprise+. The alternative was to simply buy extra sockets of Enterprise just to get the vRAM allowance but that was more expensive and got us no extra functionality.
Voila – customers are driven to the Enterprise+ tier and VMware finally have their end goal!

What a week! My son was born on Sunday and on Friday night I finally received notification that I passed the VCAP5-DCA beta exam I sat back in May. Good things come to those that wait!

I really thought I’d failed as I didn’t even attempt four or five questions and I know I missed a few bits on the ones I did answer. It’s a great relief to have it out of the way however – with vSphere6 slated for a 2013 release it means I’ve got around two years grace before I have to consider upgrading again (assuming it takes a year for the exam to become available like last time!). It’s worth noting that 37% of the visitors to my blog (who responded to my poll on the right) don’t think the VCAP exams add any value over and above the VCP so maybe I’m just ‘keeping up with the Joneses’ for the sake of vanity!

Next up for me is VCAP5-DCD, VCP-IaaS, and VCP-DT, probably in that order. It’s like a cereal pack competition where you have to collect them all…

Firstly an apology – this isn’t my usual technology based blogpost. I’ve kept my posts very focused for my target audience since starting over two years ago so I’m hoping my readers won’t mind one personal post. Some things are worth making an exception for…

I’m happy to announce that my own unique ‘cloning’ project is out of stealth mode ( ) after nine months. Zachary Grigson was born on Sunday 5th August at 14:47pm and both mother and baby are doing well! He’s our first so we’re just now embarking on a massive learning curve and I’m assured my life will never be the same again (in a good way, apart from the nappies that is).

I know this will impact my blogging ability for a while but the world won’t stop turning if that happens! Like Bob Plankers maybe the ‘time off’ will give me a valuable new perspective? I’m planning to make VMworld Barcelona but I’ll probably skip the parties in exchange for some sleep – I think I’ll need it.

With VMworld rapidly approaching I’ve been wondering what to expect from this year’s shows in San Francisco and Barcelona. My first thought was ‘not much new’ based on the fact that vSphere5 was released at last years show and products such as vCOPs and vCloud Director have already had major updates recently. Upon reflection however I’ve realised that this year will be no different – trying to keep up with all the announcements, new technology, product launches and social gatherings will be nearly impossible. First let’s have a quick recap of the last two year’s shows and the various releases;

History of VMworld announcements

• 2010 – Virtual Roads, Actual Clouds (attendance: 22000)
  US – vCloud Director, vFabric, vShield, vSphere 4.1 (announced just before the show), Project Horizon previewed
  Europe – vCloud Request Mgr, vCenter CapacityIQ

• 2011 - It’s your cloud, own it! (attendance: 25,000+)
  US – Updated ‘cloud suite’ (vSphere5, view5, SRM 5, vCD 1.5, vShield 5) announced weeks before the show, vFabric Data Director, Project Octopus, Appblast, Horizon Application Manager, VXLAN
  Europe – vCenter Operations Management suite, vCenter vFabric Application Management Suite, vCenter Protect Essentials Plus

Clearly the major launches are always at the flagship show in the US, with a few kept over to keep Europe happy.

What’s in store for 2012?

VMworld is always a mix of corporate strategy and ‘in the trenches’ technology deep dives. There’s always a long term, cohesive, persuasive story which brings all the disparate technology elements together in a compelling vision. Lots of column inches have been dedicated to the recent management reshuffle at VMware and EMC combined with commentary about the evolution

Read more…

I’m planning to attend VMworld Europe again this year and like many I’ve just started looking into accommodation. While it’ll be nice to be in Barcelona again I’ll miss Copenhagen as I had a ‘regular’ place to stay just a five minute walk from the conference whereas this year I’ll be hoping the place I’ve booked is both well located and nice to stay in.

Last year Andi Mann (@andimann) posted a very useful mashup map of the VMworld hotels, making it much easier to work out where to stay. I recall (as I think the post is no longer available) that he suggested to VMware that they provide the hotel information in a more accessible format but unfortunately it’s still only via the official webpage (or you can use my Excel data). The VMware page lists quite a few hotels and their distance from the conference but it’s far from ideal. For starters some distances are in km and some are in miles but it’s easy to miss that little detail.

Rather than putting the burden on Andi to repeat last year’s map I thought I’d have a go. Turns out the basic mapping is pretty easy to do once you have the hotel data formatted as you can just cut and paste it into BatchGeo.com.

Here’s a map of the ‘official’ VMware hotels for Barcelona 2012, colour coded by ‘star’ rating (the conference location is in red). Note that if you scroll down under the map the hotels are listed alphabetically so you can easily locate a hotel by name;

View VMworld Europe 2012 hotels in a full screen map

There are a couple of things to note. The bulk of the hotels are clustered around the centre of town rather than the conference venue so you’ll have to decide which is more important to you – proximity to the conference or to the restaurants and clubs (which inevitably host the vendor parties). More importantly take the ‘distance’ mentioned on the VMware website with a pinch of salt – BatchGeo does its own calculation on distance which shows a few discrepancies. Batchgeo does an ‘as the crow flies’ calculation and maybe VMware have driving distances?

I’ve not had time to check but last year I found it was cheaper to book my hotel directly rather than via the VMworld registration process (despite the promised discount). Your mileage may vary!

Lastly thanks are due to Andi Mann – I wouldn’t have known what to do this without his original. I look forward to seeing everyone at the conference!