Questioning Competency Assessment

Ted Metcalfe provides me (us) with much insight and deep thinking about engineering, competency, ethical behaviour, learning from (engineering/technical) failures, and all kinds of other issues and aspects of life. He’s semi-retired but thankfully not sailing off into the sunset yet.

He often talks about raising the competency/skills of up-and-coming engineers.

We both share that interest, and both want to be there to help with that, but we sort of disagree on whether, while raising that competency, we should be testing people after or along the way.

Like, I think he means, bona fide, marked, challenging, no cheating, sweat running down your back tests, like back at university. {shudder}.

I lean towards “some just aren’t good at taking tests so let’s not” and he leans to “we must test to prove competence”.

Below is a long-ish article he’s written and sent to me in an email with the subject “Controversial post to consider”. I don’t know if it’s really that controversial, but it does put a question to us. Should we be testing for competence, not just assuming it.

Preamble:

After Ted presented to the APGA community in June 2022 on “Failure is Normal: A Tale of Two Bridges” (the Quebec Bridge and the Westgate Bridge) (link to APGA webinar here), he went back and reviewed the list of similarities between the two bridge failures, and he realised that this similarity in particular represented an important message for engineers:

The designer / consulting engineer reputations went unchallenged.

The engineering firms engaged in both cases were assumed to be competent by reputation alone, but that “competency’ was not proven before selection in either case.

Below are not my words, they are Ted’s. Let’s discuss.

===========================

Questioning competency assessment

By Ted Metcalfe, Independent Consultant

In the Quebec Bridge (1907 and 1916) and West Gate Bridge (1970) failure events, more than 100 construction workers were killed partly as a result of failings by design engineers, yet the competency of those engineers was neither questioned nor proven before they undertook their design work.

Would Registration of engineers have prevented the West Gate bridge failure?

In hindsight, no one knows for sure, but I doubt it. Here’s why.

In 2018, the Victorian government introduced legislation to register professional engineers. This was actually a direct consequence of failed government regulation of the occupied building industry over previous decades; however rather than admit failure, governments prefer to identify scapegoats.

Starting about three decades ago, building regulations were progressively modified to “encourage economic activity and create jobs”; and bureaucrats created “deemed to comply” interpretations of the Building Code; all of which collectively allowed non-compliant materials and dodgy practices by developers to proliferate.

I strongly suspect that even if the Victorian Professional Engineers Registration Act had already been in place when the West Gate bridge project was undertaken, the reputation of the design engineer was such that the government of the day would simply have declared that the firm was “deemed to comply” with the competency requirements of the legislation, and the tragedy would still have happened.

In Australia today we have a serious threat to public safety. Thousands of homeowners are stuck living in fire traps because of the flammable cladding debacle, and even more are struggling to get serious building defects rectified. 

The government’s answer? …………. Make it look like building engineers are to blame!

Who says whether or not (Pipeline) Engineers are competent?

For many years our industry-specific pipeline systems standard (AS 2885) has required that it only be used by competent persons. Looking back now, I realise that engineering competency demonstration and assessment has been an issue in our industry for a long time, and it is still not resolved to my satisfaction.

About twenty years ago, some industry experts conducted a number of seminars to emphasise both competency and compliance with the AS 2885 suite of Standards. 

They included a mock “court in session” role play to put individual engineers “on trial” for failure to comply with the requirements of AS2885; or for not being “a competent person” as required by AS2885.

In one session they picked me as the suspect, and of course I was judged by His Honour Haddow in the black robes and curly wig to be “guilty” of some crime we’ve both since forgotten.

However, I clearly recall responding to his sentence by demanding to know how the average pipeline engineer was supposed to determine or demonstrate competency as required by the Standard. 

I don’t recall getting anything like a clear and concise answer back then.  

How is engineering competency assessed now?

Competency based on the APGA PECS is currently assessed by a panel of engineers who review evidence submitted and interview the applicant. “Competent” or otherwise is essentially a judgement call.

The PECS originated back in 2008, when myself and others prepared a Business Case with a focus on training for pipeline engineers.

At the time, as part of our research I contacted Phil Hopkins, the eminent educator of pipeline engineers worldwide, and from my notes of our conversation it was his opinion that….

“…. competence can only be assessed by formal examination under controlled conditions with a clearly defined level of correct answers.” 

It made sense to me back then (and still does) that in order to demonstrate that you have learned something you must be able to prove under test conditions that you have indeed retained the information.

Our recommendation for development and operation of industry-sanctioned training courses with examinations was not accepted by the Association Board at the time. Instead, a project was undertaken to develop the competency documentation (resulting in 240 competencies, the Pipeline Engineering Competencies System), for commercial entities to use as they see fit.

So now we have defined competencies for pipeline engineers which is certainly a good outcome, but the PECS approach alone is neither complete nor effective in creating change.

Why is competency demonstration not mandatory?

In my APGA Convention paper last year, “Pipelines and Public Safety: How Reliable Are We?”, I pointedly observed that even with the PECS in place, competency demonstration is still not mandatory for engineers in our own industry……. Why not?!     

How should competency be assessed?

In my opinion, competency assessment should include an examination with a minimum pass mark.

I agree that competency assessment should also include a review of written submissions and an interview with a panel, but I strongly believe we should add a written examination to the process to ensure that the assessment includes a measurable component which is not simply a judgement.

Think about it………in order to obtain a driver’s license in Australia you must pass a written examination; but to practice engineering, as long as you have degree qualifications and some experience, neither written test nor license are required.

In his book “The Making of an Expert Engineer”, James Trevelyan makes some good points about competency assessment, and from the notes I made while reading his book:

A formal examination is a necessary part of training and motivates learning.                         

No exam means there is no motivation to really make an effort to learn.

The Canadian requirements for engineering competency assessment were prompted in part by the Quebec bridge failures over 100 years ago, and those still include a written multiple-choice Professional Practice Examination

Visitors wishing to access the workplaces of most major operating companies in our own industry are required to undergo a formal induction process comprised of an on-line review of workplace safety and other corporate policy information, followed by a multiple-choice quiz to identify whether or not they have satisfactorily understood the important relevant information. 

They must pass the quiz to be allowed on site, and usually must sign something to acknowledge that they have understood the information.

Can we implement exams for (Pipeline) Engineers?

We can. 

With our PECS already defined, much of the hard work has already been done; it’s just our competency assessment process that needs changing.

The pipeline industry must voluntarily and formally recognise pipeline engineering competency at least as highly as visitor safety in the workplace. 

If we do not, in the event of a serious accident our regulators may well impose upon us a legal “duty of care” to do so. 

As I said in my recent webinar, and as demonstrated above for the occupied building industry, engineers are an easy target for blame.

What do you think?

Surely someone else out there is willing to express an opinion on a matter this critical for our industry.

Please consider this carefully, and you are encouraged to leave a reply either supporting or objecting to my positions. 

If you are opposed to adding a written competency examination to the assessment; or if you do not agree that we should make competency assessment mandatory for pipeline engineers, please reply to join the conversation and explain your positions.

Even if you don’t have an opinion yourself, just forward this to someone else you know who should.

I always want to encourage debate about standards, ethics, and our industry. Every opinion matters.

Ted Metcalfe, Independent Consultant

Book Club for Failures

There is enough interest in Ted Metcalfe’s book list, and the idea of learning from failures, that it’s time to set up a book club. It’s a book club for people in the pipeline engineering industry, interested in learning from failures. Answering the question “what happened?”, and being better engineers and pipeliners because we’ve paid attention to the lessons.

In a “book club” type of environment, we can assign ‘required reading’, but more importantly, there is the opportunity to discuss what we’ve read, learned or thought about each month. That’s what I’m looking forward to.

The first meeting, a kickoff meeting to check interest and participation, will be held on Monday August 1st , 16:30 – 17:30 AEST, via either Zoom or MS Teams.

Put it in your calendar for now. In the next week or so, I’ll set up a signup link and meeting invitations to confirm interest. That will appear here on the blog.

The “book” we’ll discuss at this first kickoff meeting is the APGA Engineers Practice Guide.

Unplanned Outcomes, Stories, and the Intent of our Standard

by Ted Metcalfe

It may have been Einstein who famously said “The more I learn, the more I realise how much I don’t know.

More simply put, “We don’t know what we don’t know”.

Photo by Iva Muu0161kiu0107 on Pexels.com

Gaining awareness of things which have not turned out well for others helps us to get better at what we do so that we don’t repeat mistakes others have made.

Industry codes and standards record the wisdom and experience accumulated by many engineers over many years for the benefit of all engineers, and these need to be updated regularly for various reasons including knowledge of things which have not turned out well.

In his book The Making of an Expert Engineer, Prof. James Trevelyan states at Page 57:

Technical standards have been created through the experience of other engineers and are carefully negotiated within each specialised engineering discipline, striking a balance between restrictions to promote safety and ease of use, while also avoiding constraints that would inhibit innovation and design freedom.

That’s an excellent description of what the engineers on our Standards committees do all the time.

In the Australian pipeline industry we have an excellent Standard (AS 2885) to work with because we revise it regularly to keep up to date with new information.  Awareness of poor outcomes is one of the things that has informed the continued evolution of the Australian Standard in a way that pipeline engineers in other countries envy. 

Here’s some examples of links between past events and the current Standard:

Part 1 Clause 13.2 (a) requires project design records include as-built data. 

Example: Due to a contractual issue, the as-built survey data for an HDD was not provided to the client. A few years later, during third-party works near the HDD, the original design drawings were assumed to be accurate. They were not. The third-party works punctured the pipeline far below the surface, and the client ended up replacing the entire HDD string.

Photo by Pixabay on Pexels.com

The AS2885 suite requires detailed attention to SCC and Fracture Control.
In 1982, a major rupture of the Moomba Sydney Pipeline over some distance prompted a lot of research and investigation associated with Stress Corrosion Cracking and the importance of Fracture Control to arrest running fractures. The outputs of the research and investigations have resulted in revisions to AS 2885 in several areas.

Photo by Monstera on Pexels.com

AIV and FIV (Vibration)
AS2885.1 now includes clear delineation between “linepipe” and “piping”, and makes specific reference to AIV and FIV as potential failure modes. The unexpected discovery of an integrity threat on a relatively new pipeline system, and the research associated with mitigation of that threat, have now been incorporated in to AS2885 as revisions in several areas including the vibration Appendix.

Photo by Anni Roenkae on Pexels.com

Knowledge of such events leads to a better understanding of the intent of the Standard, but sometimes the background for changes in the Standard is not widely known by those who use it.

Sharing of stories about things which have not turned out as planned is one way to increase awareness for better understanding of the intent of the Standard.

Sometimes that requires sharing stories with others about things that we ourselves have not done well, which can be embarrassing.

Despite the reluctance to share such stories, the AS2885.info team and others believe that we can and should get better at helping others learn through sharing stories.

If you are intrigued by the concept of sharing stories to help others better understand what went wrong and avoid making the same mistakes, please contact us.

Gaining Confidence

by Ted Metcalfe

Do I know what I’m talking about?

Photo by Rodrigo Souza on Pexels.com

Experienced engineers are able to make engineering judgements with confidence. Some of the reasons why pipeline engineers using AS2885 may benefit from asking a question in relation to confidence include:

1) Maybe you are required to make a decision in relation to application of the Standard, but just don’t quite have the confidence to do so, and a second opinion would help.

…I don’t know enough about this, but I’ll bet someone else around here does…

2) Maybe you have been told by your supervisor or an experienced colleague that a certain clause means one thing, but their interpretation does not seem quite right to you, and you would like a second opinion without openly challenging your colleagues.

…that’s not what I think it means; I need more guidance here…

3) Maybe you have witnessed what you think might be an incorrect or inappropriate practice, and before making any fuss about it in your own workplace you would like to quietly get a second opinion from an independent source, without disclosing why you are asking.

…I’m pretty sure this isn’t right, but I need confirmation…

4) Maybe you are involved with modifications to a rather old pipeline for which not all of the usual design and inspection material is available, and you are unsure as to exactly how the current Standard should be applied.

…this pipeline is older than me, and it needs help… and so do I…

5) Maybe you are afraid that your question will be considered by other as a dumb question, and you don’t want to ask in the office and risk looking silly for not knowing the answer already.

…I’m not dumb, but I feel that way…

This last point prompts me to describe how I learned a very important lesson about asking questions quite early in my career when I was working in a sour gas processing plant:


Amoco Canada processed a lot of highly toxic hydrogen sulphide gas and the gas plant where I was working had experienced a serious accident. As a very junior engineer I was allowed to attend the meeting of management convened to examine the causes and work out a way forward, but I didn’t understand everything that was being discussed.

At one point I bravely put up my hand and said “Can I ask a stupid question?”

The plant manager replied calmly from the other end of the table “Son, in this industry, there are no stupid questions, only dead people who failed to ask the questions, so how can we help you?”


Ever since then I have had the confidence to ask a question when I didn’t understand something.

You can ask the AS2885.info team any questions which might help you be a better pipeline engineer – that’s what we’re here for.

Asking Questions is the Easy Way to learn

Be a better pipeline engineer – ask the question!

by: Ted Metcalfe

Photo by Ksenia Chernaya on Pexels.com


We learn new information in many ways, and for many different reasons. Even when we are not trying to learn, or don’t think we need to learn, we seem to gather valuable information.

For some people, lessons are really only learned if they are learned the hard way, from the bitter experience of having done something wrong with unexpected (sometimes embarrassing, painful or expensive) consequences.

It’s a lot easier to learn by asking questions.

In the Introduction of Part 0, it is acknowledged that AS2885 sets out specific requirements in some areas, but notes that these do not replace the need for appropriate experience and engineering judgement.

In Clause 1.5.7 of AS2885 Part 0, competence is defined as having an appropriate combination of knowledge, skills and experience to safely and effectively perform the task required as requirements for users of AS2885.

Users of AS2885 are required to apply engineering judgement.

It can be said that engineering judgement requires a combination of both knowledge and the confidence to make decisions, where:

– Knowledge is the accumulation of relevant factual material, and

– Confidence is the self-courage required to interpret both the relevant circumstances being considered and the application of the Standard to those circumstances, and to make decisions on that basis.


Experienced engineers have learned that one of the easiest and most important ways to learn is to simply ask questions, and for pipeline engineers, that’s one of the main reasons that the AS2885.info wiki was created.

We’re here to help you learn, and we look forward to having users of our Standard ask questions.

Accumulating Knowledge


by Ted Metcalfe.

Ask the question!

Photo by Pixabay on Pexels.com

Some of the reasons why pipeline engineers may benefit from asking a question in relation to accumulating knowledge include:

1) Maybe the matter you are working on is a bit out of the ordinary, and you are not sure exactly which part of the Standard should apply.

….Which clause covers this?

2) Maybe a clause in the Standard that you were previously familiar with has been changed or has disappeared, and you want to know what clause applies instead.

….Where’s that clause gone, what do I do now?

3) Maybe you have learned about some innovative new product or material that does not seem to be covered already in the Standard.

…Can I use this new-fangled approach?

4) Maybe you have an innovative idea that you would like to implement, but to do so might be stretching the intended scope of the Standard.

…Can I change the way we do this and still conform to the Standard?

5) Maybe the wording of a clause seems confusing, or could be interpreted two different ways.

…whaaaat?

6) Maybe you simply want to learn more about a particular topic in pipeline engineering, construction or maintenance, but don’t know where to look for additional information.

…I’m new here, where do I start?


This last point illustrates that while good experienced engineers always work within their limits, those limits can be expanded by additional training, study or participation in events.

If the AS2885 team can’t help directly with any of the above, someone in their wide networks of industry contacts probably can, and we will make the introductions for you.