Questioning Competency Assessment

Ted Metcalfe provides me (us) with much insight and deep thinking about engineering, competency, ethical behaviour, learning from (engineering/technical) failures, and all kinds of other issues and aspects of life. He’s semi-retired but thankfully not sailing off into the sunset yet.

He often talks about raising the competency/skills of up-and-coming engineers.

We both share that interest, and both want to be there to help with that, but we sort of disagree on whether, while raising that competency, we should be testing people after or along the way.

Like, I think he means, bona fide, marked, challenging, no cheating, sweat running down your back tests, like back at university. {shudder}.

I lean towards “some just aren’t good at taking tests so let’s not” and he leans to “we must test to prove competence”.

Below is a long-ish article he’s written and sent to me in an email with the subject “Controversial post to consider”. I don’t know if it’s really that controversial, but it does put a question to us. Should we be testing for competence, not just assuming it.

Preamble:

After Ted presented to the APGA community in June 2022 on “Failure is Normal: A Tale of Two Bridges” (the Quebec Bridge and the Westgate Bridge) (link to APGA webinar here), he went back and reviewed the list of similarities between the two bridge failures, and he realised that this similarity in particular represented an important message for engineers:

The designer / consulting engineer reputations went unchallenged.

The engineering firms engaged in both cases were assumed to be competent by reputation alone, but that “competency’ was not proven before selection in either case.

Below are not my words, they are Ted’s. Let’s discuss.

===========================

Questioning competency assessment

By Ted Metcalfe, Independent Consultant

In the Quebec Bridge (1907 and 1916) and West Gate Bridge (1970) failure events, more than 100 construction workers were killed partly as a result of failings by design engineers, yet the competency of those engineers was neither questioned nor proven before they undertook their design work.

Would Registration of engineers have prevented the West Gate bridge failure?

In hindsight, no one knows for sure, but I doubt it. Here’s why.

In 2018, the Victorian government introduced legislation to register professional engineers. This was actually a direct consequence of failed government regulation of the occupied building industry over previous decades; however rather than admit failure, governments prefer to identify scapegoats.

Starting about three decades ago, building regulations were progressively modified to “encourage economic activity and create jobs”; and bureaucrats created “deemed to comply” interpretations of the Building Code; all of which collectively allowed non-compliant materials and dodgy practices by developers to proliferate.

I strongly suspect that even if the Victorian Professional Engineers Registration Act had already been in place when the West Gate bridge project was undertaken, the reputation of the design engineer was such that the government of the day would simply have declared that the firm was “deemed to comply” with the competency requirements of the legislation, and the tragedy would still have happened.

In Australia today we have a serious threat to public safety. Thousands of homeowners are stuck living in fire traps because of the flammable cladding debacle, and even more are struggling to get serious building defects rectified. 

The government’s answer? …………. Make it look like building engineers are to blame!

Who says whether or not (Pipeline) Engineers are competent?

For many years our industry-specific pipeline systems standard (AS 2885) has required that it only be used by competent persons. Looking back now, I realise that engineering competency demonstration and assessment has been an issue in our industry for a long time, and it is still not resolved to my satisfaction.

About twenty years ago, some industry experts conducted a number of seminars to emphasise both competency and compliance with the AS 2885 suite of Standards. 

They included a mock “court in session” role play to put individual engineers “on trial” for failure to comply with the requirements of AS2885; or for not being “a competent person” as required by AS2885.

In one session they picked me as the suspect, and of course I was judged by His Honour Haddow in the black robes and curly wig to be “guilty” of some crime we’ve both since forgotten.

However, I clearly recall responding to his sentence by demanding to know how the average pipeline engineer was supposed to determine or demonstrate competency as required by the Standard. 

I don’t recall getting anything like a clear and concise answer back then.  

How is engineering competency assessed now?

Competency based on the APGA PECS is currently assessed by a panel of engineers who review evidence submitted and interview the applicant. “Competent” or otherwise is essentially a judgement call.

The PECS originated back in 2008, when myself and others prepared a Business Case with a focus on training for pipeline engineers.

At the time, as part of our research I contacted Phil Hopkins, the eminent educator of pipeline engineers worldwide, and from my notes of our conversation it was his opinion that….

“…. competence can only be assessed by formal examination under controlled conditions with a clearly defined level of correct answers.” 

It made sense to me back then (and still does) that in order to demonstrate that you have learned something you must be able to prove under test conditions that you have indeed retained the information.

Our recommendation for development and operation of industry-sanctioned training courses with examinations was not accepted by the Association Board at the time. Instead, a project was undertaken to develop the competency documentation (resulting in 240 competencies, the Pipeline Engineering Competencies System), for commercial entities to use as they see fit.

So now we have defined competencies for pipeline engineers which is certainly a good outcome, but the PECS approach alone is neither complete nor effective in creating change.

Why is competency demonstration not mandatory?

In my APGA Convention paper last year, “Pipelines and Public Safety: How Reliable Are We?”, I pointedly observed that even with the PECS in place, competency demonstration is still not mandatory for engineers in our own industry……. Why not?!     

How should competency be assessed?

In my opinion, competency assessment should include an examination with a minimum pass mark.

I agree that competency assessment should also include a review of written submissions and an interview with a panel, but I strongly believe we should add a written examination to the process to ensure that the assessment includes a measurable component which is not simply a judgement.

Think about it………in order to obtain a driver’s license in Australia you must pass a written examination; but to practice engineering, as long as you have degree qualifications and some experience, neither written test nor license are required.

In his book “The Making of an Expert Engineer”, James Trevelyan makes some good points about competency assessment, and from the notes I made while reading his book:

A formal examination is a necessary part of training and motivates learning.                         

No exam means there is no motivation to really make an effort to learn.

The Canadian requirements for engineering competency assessment were prompted in part by the Quebec bridge failures over 100 years ago, and those still include a written multiple-choice Professional Practice Examination

Visitors wishing to access the workplaces of most major operating companies in our own industry are required to undergo a formal induction process comprised of an on-line review of workplace safety and other corporate policy information, followed by a multiple-choice quiz to identify whether or not they have satisfactorily understood the important relevant information. 

They must pass the quiz to be allowed on site, and usually must sign something to acknowledge that they have understood the information.

Can we implement exams for (Pipeline) Engineers?

We can. 

With our PECS already defined, much of the hard work has already been done; it’s just our competency assessment process that needs changing.

The pipeline industry must voluntarily and formally recognise pipeline engineering competency at least as highly as visitor safety in the workplace. 

If we do not, in the event of a serious accident our regulators may well impose upon us a legal “duty of care” to do so. 

As I said in my recent webinar, and as demonstrated above for the occupied building industry, engineers are an easy target for blame.

What do you think?

Surely someone else out there is willing to express an opinion on a matter this critical for our industry.

Please consider this carefully, and you are encouraged to leave a reply either supporting or objecting to my positions. 

If you are opposed to adding a written competency examination to the assessment; or if you do not agree that we should make competency assessment mandatory for pipeline engineers, please reply to join the conversation and explain your positions.

Even if you don’t have an opinion yourself, just forward this to someone else you know who should.

I always want to encourage debate about standards, ethics, and our industry. Every opinion matters.

Ted Metcalfe, Independent Consultant

8 thoughts on “Questioning Competency Assessment

  1. couldn’t agree more that at least a written competency examination will be required for practicing pipeline engineers. I suggest that a competency book be put together by experts in APGA in order for pipeline engineers to study and take part in the exam

    Liked by 1 person

    • Thanks for your support on this proposal.
      Your suggestion is a good one. Preparation of a Study Guide for Pipeline Engineering Competency Assessment to help applicants prepare for the exams would be an important addition to preparation of the exams themselves.
      We will keep this in mind as the discussion continues.

      Like

  2. This is complex, because pipeline engineering is complex. I can certainly see the merit in an exam but there are so many competencies to be assessed that developing exams to adequately cover every aspect of pipeline engineering competence would be a massive task. Developing the competency descriptions themselves was a huge project, not because each was long or difficult but because of their sheer number (I know because I was involved in several of them). But writing a brief outline of a competency sounds easy compared with writing an examination paper that will seriously but fairly test a candidate. Not to mention marking the exams. Does the industry have the people and resources to implement this?

    Like

    • Yes, pipeline engineering is complex, but in my opinion that does not justify taking an easier line on competency assessment.
      Driving is also a complex skill, but driver examinations are conducted only on the basics, not on the complexities. Recovery from skids, driving in darkness and inclement weather, or towing a trailer are not tested, but a full license allows the driver to undertake those activities.
      About 40% of first-time attempts by learners fail the driver’s license examination. Imagine the carnage on our roads if licenses were awarded only on the basis of an interview, references from other drivers and watching a video of the applicant successfully piloting a car down the road.
      Currently, assessment of the competency of a pipeline engineer relies heavily on the judgement of those individuals conducting the assessment. Adding an examination to the procedures would reduce this reliance on personal judgements, adding transparency to the outcome.
      Finally, your last question is perhaps most important in this discussion for two reasons:
      • Firstly, in a coroner’s court, I expect that the prosecution lawyers would seize upon the contrast between the massive volume of effort in developing our PECS and the absence of any obligation to be assessed as competent in order practice as a pipeline engineer in our industry. Not a good look.
      • Secondly, our industry has traditionally failed to adequately value the contributions of highly experienced independent consultants who serve on a volunteer basis for the wider benefit of the industry. Perhaps if that matter were fairly and equitably addressed, we would have no difficulty finding the resources to get the examinations prepared and marked.
      More comments from others are welcome.
      Ted Metcalfe

      Like

  3. Competence is one aspect, however invariably, but especially in operating and maintaining pipelines, because one rarely (if ever) has all the information, good engineering judgement comes into play.
    In my view good engineering judgement is accumulated with wide experience, recognizing when more information is essential or when specific input from those with more knowledge is required.

    I guess in future AI will deal with competence but I’m not sure how you measure good engineering judgement and I’d welcome comment on how you might test this aspect?

    Like

  4. I’m not convinced that AI will deal with competence, but I agree with your description of good engineering judgement. I don’t immediately have a definitive answer as to how judgement might be measured by examination, and others are welcome to make suggestions.
    Judgement requires decisions, and before making any decision an engineer exercising good judgement would ask:
    • Is a decision actually required?
    • Is this matter outside my area of competence?
    • Do I have enough information, experience and the confidence to make a decision?
    How we think, and how we recall past experiences and make decisions is one of the most complex topics I’ve read about in books by a number of prominent authors, notably Daniel Kahneman, Gary Klein and Malcom Gladwell.
    I will suggest that:
    a) Good judgement would be demonstrated by making a similar decision to that which the majority of experienced engineers would have made in the same situation, and
    b) Whether or not one has exercised good judgement can only be assessed by others.
    That’s why we need a large group of experienced engineers to help set the exams, work out questions which would allow assessment of judgement, and determine what would be acceptable answers to each of the questions.
    I further suggest that this would probably involve multiple-choice questions and there may well need to be ethical issues addressed in some cases.
    Comment is welcome from others who can offer other suggestions about examining capability to make good engineering judgments.

    Like

  5. While a written competency demonstration examination makes sense; taking lessons from API certification procedures where Pressure Vessel and piping inspectors are certified after passing a written examination; I have seen numerous examples where the exam does not translate into real knowledge & we have several “certified” personnel who cannot pass the rudimentary requirements.

    So, I guess, competency demonstration is more of a judgement call based on knowledge & experience. API have exacerbated this issue by charging renewal fee every 3 years and made this into a money making scheme.

    Like

    • You make some good points here, each worthy of a response:
      1) It is easy to identify examples of assessment failures in any occupation, but that does not prove that the assessment system applied is at fault. I have studied documentation prepared by engineers who had been assessed as competent without formal testing, and found serious flaws in their work.
      2) Yes, competency assessment is a judgement call based on knowledge and experience. Currently, those doing the assessment rely upon their own knowledge and experience to judge the knowledge and experience of the applicants, an approach which can be made more robust and reliable with a tangible test result as evidence. In my opinion, both experienced judgement based on review of applicant submissions and a formal examination should be applied in assessing competency.
      3) Thanks for focusing on the rudimentary requirements. Others have expressed concern about creating exams able to test all of the complexities and expertise levels of pipeline engineering, and I hasten to clarify that I am proposing testing of the rudimentary, entry-level requirements only.
      4) API is certainly not alone in playing the revenue game, as some organisations seeking memberships of engineers in Australia have also demonstrated. To my knowledge, APGA is an exception on this matter, as the assessment work is done by experienced volunteers.
      More comment from other readers is welcome.
      Ted Metcalfe

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s