Blog

Rogers Communications is a private company that provides phone and internet services (as well as cable TV in some jurisdictions).  No customer is obligated to choose Rogers and there are alternative suppliers for these services available to most Canadians.

On July 8, 2022, the internet and cellular phone networks operated by Rogers in Canada went down.  Few people notice when one of our servers is shut down for a day.  When Rogers goes down for an hour the country is in disarray.  Beyond inconvenience, people that have abandoned land lines lost access to 911 and some emergency call centers went down as well.  Catastrophic for small business the entire Interac payment system went down leaving some retailers unable to accept customer payments.  In a nutshell it was ugly.

What happened?  It is a technical story about routers and DNS servers – I would best summarize the story as ‘someone made a boo boo’.  But there is a deeper question:  how in a publicly traded organization with highly paid executives and highly experienced directors did a company allow a single point of failure to collapse the entire network?  I don’t have answers to this question, but we now know that the company CTO lost his job – someone had to wear the nametag of the failure and it is rarely the CEO.  Someone at Interac should lose also their job for not realizing that having a single internet provider was a single point of failure.  Even tiny ASI has a backup internet supplier.

But to blame the individual in the tech department that implemented the system change that led to the blackout or even blaming the CTO misses the point.  The failure is owned by the CEO and the Board of Directors.  Those parties don’t need to vet every decision, but they do need to oversee the structure of the business redundancies and safeguards.  To me, blaming someone in purchasing that a key insurance policy wasn’t renewed ignores that there should be a reporting system all the way to the Board of Directors that identifies the insurance needed and confirms that the insurance has been purchased.  Redundant internet is just another form of insurance.

Quality management processes are not new ideas.  Twenty years ago, on August 27, 2002, we had our first audit of ASI’s newly designed Quality Management System.  For the past 20 years, it has been a rewarding journey, into the world of quality and the International Organization for Standardization.

Why?

We started ASI in 1998 and by 2002 we were just two people – one pension administrator and one actuary.  We had adopted many of the best practices we learned at Mercer, Aon, and Towers.  In particular, we adopted the cardinal quality control measure of having a second person check the work of the first.  But at that time we didn’t have much in the way of written procedures – everything relied on our personal experience.

My original thinking was that we needed to document the procedures for administering the pension plans of our clients so that we wouldn’t have to rediscover the correct calculation each time we did one and ultimately when we hired new team members they would acquire our experience without needing to live our lives.  My goals were both accuracy and efficiency.

What?

What started out as an effort to document how we administered pension plans grew into a full blown ‘system’ for running our business.  Consulting, actuarial, and pension administration all had their processes documented.  Beyond that, the process of making improvements – whether identified proactively or as a result of an error was embedded in the system, training was codified, and qualifications to sign off on work were established.  Finally, the entire system was wrapped in ‘management oversight’ where we meet quarterly (originally annually) to review our quality metrics which focus primarily on being on-time (virtually 100%), error free (virtually 100%) and happy clients (exactly 100% for many years now).

How?

The story behind our quality system is as random as any.  I was playing pick-up hockey in Oakville in January 2002, and one of the guys in our group showed up relieved to have ‘survived’ his ISO audit.  He was in manufacturing and audits were always difficult days where auditors would find flaws in the system or times where staff did not follow the already well-designed system.  As he shared his story, he said that they would never make it without their consultant John.  A week or two later I was talking to John about how he could help us build our own ISO compliant management system.

In retrospect, the cost of hiring John was modest.  If you told me how much time Paula and I would spend that first year on nights and weekends writing out everything I would never have agreed to proceed, but like so many times in my life (notably actuarial exams), by the time I realized there might be a better path I am past the point of no return.  Eight months after we started the project, we received our registration as compliant with the ISO 9001:2000 quality standards.

Evolution

Done properly, ISO registered systems continuously renew themselves to capture new thinking, learning from mistakes, and changes in business conditions.  This ‘continuous improvement’ mindset is at the core of our organization.  As staff join the team, they are indoctrinated into our system and culture where everyone plays a supporting role in keeping the system up to date.  Most of our procedures have been edited at least once or twice since they were first designed.  For some there have been dozens of changes as we chase after changing laws and actuarial standards.

For most of our clients, the system is most visible when we present our client survey in the fall and our annual planning memo at the end of the year.  The less visible part of the system is the absence of the need to call us to ask where we are with a project or if we will be on-time.  Clients often don’t notice when you don’t make mistakes and they aren’t calling you to fix problems.

It has been at least a decade since Paula and I reviewed every document that left our office.  We have entrusted the quality of our product to the passionate members of our team who take as much pride as we do in getting the work done right while relying on the system to give everyone comfort that if they follow the process the right outcomes will follow.

Today

Internet service and access to smart phones is no longer a luxury – it has become a basic requirement for almost every business, student, and individual that wants to participate in our society.  When this outage occurred, Rogers was already in the process of trying to merge with Shaw Communications – a competitor in the same space.  I am a laissez fair kinda guy and I am never excited to see government trying to interfere too much with private business.  Because of these events, Canada’s government has been asking tough questions about whether this merger should go ahead – in this case maybe appropriately so.  I don’t have the underlying business case for the Shaw deal so it’s hard for me to have an opinion on what the government should do.

To me this system failure couldn’t come at a worse time for Rogers because maybe the truth is that they shouldn’t be allowed to merge with Shaw, not because consumer pricing would get worse, but because these services are too critical to Canadians and we cannot afford to have all our eggs in one basket – especially when the control of that one basket is vested in a single individual.  I am sure Rogers’ reply will be that everything will be fine – it will be interesting to see how the government decides this one.  If someone important at Rogers is reading this my name might come off the list of consultants that they want to hire.  But maybe the fact that I am prepared to point out what they need to be doing differently is exactly why they should be calling upon me.  I won’t hold my breath.