comment 0

The Problems with Task Based Performance Metrics in IT – Updated

Task based performance metrics, at their root level, measure someone’s performance based on quantity instead of quality. The attraction is the ease of counting the tasks completed, but there are problems caused by using this type of metric in IT.

What are the problems with task based performance metrics in IT?

 

It devalues harder tasks and tickets that required extra troubleshooting unless you alter the metrics to give extra weight to more complex tasks – or leads to creation of extra tickets for the extra work to get credit.

 

Alternatively, it creates segregation of first, second and third level technical support so that subject matter experts are held to a different standard than general tech support. It results in rote tasks to be completed more quickly by people who know less overall; this limits their ability to handle more complex matters and ones easily mistaken for something else properly.

 

Relying on the number of tickets as a measure of performance hurts customer service when it drives employees to close things as quickly as possible, though customers may need more handholding.

 

Teams may look at automating repetitive tasks to speed them up instead of simplifying, streamlining and improving the overall process so fewer tasks are needed.

 

The shift to task based metrics and segregation of first level support encourages development of troubleshooting scripts, as well as hiring lower skilled people for a call center to solve issues based on the script quickly. Service levels go down, while the same volume of tickets is closed more cheaply.

 

There is an incentive to send someone a how to document via a link and say you’re done, regardless of how little help it is for the customer. Hey, they called back, another ticket! Constantly resetting someone’s password becomes a positive experience for the help desk, a familiar task to complete as compared to taking 10 minutes more to find out why it is continually necessary.

 

Preventative maintenance is neglected in this type of environment when system administrators’ performance is tracked in this way. When it remains a priority to management, the goals focus on how many patches, how many servers or the time it took to complete, not necessarily the quality of the work.

 

When the number of software defects found in testing is the metric by which performance is measured, expect to see every little bug reported and even gaps in documentation listed as bugs to get the count up.

 

When the number of software test steps completed or the percentage complete is the metric, you can end up with focus on getting tests completed regardless of their importance to the critical functions of the software. “We ran through 50 reports without error. Sure that data import failed, but look how we’re 95% through!”

Advertisements
comments 6

TRIZ, the Russian Problem Solving Methodology

I was asked by someone if TRIZ was the Russian equivalent to Six Sigma or Lean or some other continuous process improvement (CPI) methodology. My response was, “What is TRIZ?” And given all the various CPI acronyms out there, why hadn’t I heard of this one?

 

What Is TRIZ?

 

TRIZ is short for Theoria Resheneyva Isobretatelskehuh Zadach, Russian for Theory of Inventive Problem Solving. Translated to English, it is sometimes abbreviated to TIPS. The theory is credited to Genrich Altshuller and comrades; this is not a pun, since the man was working in the Soviet Union. And like many in the Soviet Union, he ended up in a labor camp, though since he was a preteen at the time, it didn’t prevent him from a professional career later in life.

 

The general process of TRIZ is:

  • Define your problem.
  • Compare the problem to the world’s existing problems.
  • Search through generic solutions for this type of problem.
  • Apply one of these solutions to your problem.

 

Is TRIZ an Iterative Process?

 

No, TRIZ by itself is a one-time solution for each problem that arises. TRIZICS tries to build on TRIZ, adding non-TRIZ problem solving tools and turning TRIZ into an interactive process like Six Sigma, Lean, LSS and other continuous process improvement methodologies.  The ideal solution in TRIZ is a product that is never defective, as simple as possible and generates no waste. This is the same ideal industrial engineers hope to achieve for their products and operations.

 

The Benefits of TRIZ

 

  • Searching for an existing solution may be faster than reinventing the wheel.
  • If you find a cross-industry solution you can implement, it is more likely to succeed due to prior success elsewhere than reinventing the wheel.
  • When a cross-industry solution exists, the physical tools, chemicals or processes are easier to implement than developing your own.
  • If people are regularly mining your knowledge base, you will probably see more knowledge application across many disciplines and departments as well as the lessons learned actually learned by more than the original project team.
  • The methodology’s “contradictions” matrix identifies the most common contradictions any engineer may face so you can clearly define the problem at a “world” level like weight versus strength.
  • The 40 Inventive Principles by the methodology’s creator is a clear starting point for discussions on “How could we make our product better?”

 

The Downsides of TRIZ

 

  • If you don’t have access to the data to find a good solution, whether due to paywalls blocking information in industry publications or lack of flow of information between nations, you cannot use the best solution or even a good one.
  • If you find an existing solution you can identify and implement, it may work, but it may not be the best solution.
  • No, not every problem has already been solved in the world.
  • It aims to solve one problem at a time, and while it may solve the most urgent one, it may not resolve deeper root causes.

 

Observations about TRIZ

  • TRIZ doesn’t work as well on average with non-technical problems, though it is attempted as such. One downside of TRIZ is the relative lack of case studies on the technical problems to which it has been applied.
  • While the research driven methodology encourages using solutions already documented, it doesn’t push for documenting the before and after states of the project to prove the degree of improvement.
  • While there are TRIZ resources in English, learning the methodology’s supporting concepts is going to take time. How this rivals to learning probability distributions and statistics for Six Sigma and data analysis tools for other CPI depends on the individual.

 

Why Hadn’t I Heard of TRIZ?

 

By the time the West had the opportunity to learn about TRIZ in greater detail and take it seriously, we already had a number of continuous improvement methodologies originating both in the United States and Japan.

The benefits I can see us deriving from TRIZ today are the potential application of it in accelerating product improvement and an increased, systemic application of solutions across industries.

Quality Has a Quality All Its Own, the Implications for IEs

I heard the statement that quantity has a quality all its own. This quote has been attributed to several major figures, but the rebuttal “quality has a quality all its own” matters more to me as an industrial engineer. What are the implications of quality having a quality all its own and being more important than quantity?

 

Quality Matters – at Times – More than Quantity

 

Prior to World War 1, the default assumption was that numbers won the battle. Assuming that a large number of soldiers will overwhelm an enemy is only logical if the other side doesn’t have superior airpower, firepower or some other advantage that renders the numbers moot by converting most of them into casualties.

Guerrilla warfare in Vietnam and Islamic terrorism striking a few high profile targets to strike terror into the hearts of billions to scare them into submission have shown that having the bigger army clearly doesn’t win unless you have the numbers and strategy to beat them. Thus quality beats quantity most of the time in modern warfare.

Quality also matters more in far more mundane matters. You can see this in marketing where a high quality, precisely planned SEO strategy beats link spamming and blanket advertising. Seeking to produce items in the greatest quantity regardless of quality can kill a product when you have to recall them due to manufacturing defects or no one buys them regardless of their low price because they fall apart too soon. Quality therefore matters more than quantity in many applications, and a well-planned, high quality strike is more likely to win in many fronts.

 

Quality Becomes Reinforced by Quality as the Standard

 

When quality becomes ingrained in a corporate culture, it becomes the standard by which the organization measures itself. Employees check for defects and report issues without quality control having to closely monitor everything. When the goal is “better and better”, you are more likely to see feedback and suggestions from line staff and employees at all levels on how to make things better. This ranges from process improvements on the shop floor to shipping and receiving to billing and accounting. You end up with a corporate culture whose focus is improvement, and this is why so many focus on changing corporate culture instead of simply lecturing people on the importance of quality in one afternoon before implementing a new quality management system.

 

Quality Creep

 

Quality in and of itself is relative. Imagine that a company sets a quality standard and now says it has a high quality product. Sometimes the bar gets set for an even higher quality standard, be it less variability or a more perfect finish. Or the definition of quality shifts to higher customer service or faster delivery.

In some areas, this is beneficial to the bottom line in improving customer retention or allowing you to charge a higher price for a “better” product without much more effort. In other regards, it can result in seeking better and better end results without a direct correlation between the cost of this higher quality and the price customer may or may not pay for it. To paraphrase Scott Adams, beyond actual customer requirements, quality is a luxury you cannot afford.

 

Summary

 

Quality in the modern world often beats quantity, especially when applied strategically. Quality can become a reinforcing part of company culture, literally making it better and better. Yet quality “creep” can detract from other goals like saving money and improving productivity, so it cannot be seen as the end all, be all of the organization.

Mark Twain’s Classic Advice and Industrial Engineering Today

I heard an excellent quote I did not know came from Mark Twain: continuous improvement is better than delayed perfection. I’ve written here and elsewhere how  incremental improvements taken a step at a time leads often lead to greater progress than major shifts due to the high failure rate of “big” projects. To my surprise, many more of his other insights apply to industrial engineering today.

 

Samuel Clements / Mark Twain wrote, “The secret to getting ahead is getting started.” Waiting for the perfect opportunity or ideal design, like so much in life, results in no progress at all because there is no such thing as perfect. Paralysis by analysis, too, leads to one never getting started. This is why I advocate the incremental improvement method – you can start now instead of waiting for budgets and scheduled time that may never appear.

 

“Facts are stubborn, but statistics are more pliable.” There are far more commentaries than should be on clickbait headlines that trumpet the opposite of the results of the study being discussed by the article. Less commonly discussed is how statistics are often warped to fit the results one wants or decrease the negative trend one doesn’t want others to see. The difference between excluding one rogue data point and a few can lead to a literal slippery slope.

 

There’s another version of the above quote by Mark Twain. “Get your facts first, then you can distort them as you please.” Knowledge based management requires collecting accurate data and applying the meaning to it; the latter step is open to interpretation and human error.
“Few things are harder to put up with than the annoyance of a good example.” Just as one unusual case leads to bad legislation, one “classic” example can lead to mistakes because it is held up as the perfect paradigm. One perfect project methodology could be replicated in other departments or lessons learned applied to other companies, but it should not turn into a hammer that gets applied when one truly needs a saw.

 

“Don’t let schooling interfere with your education.” I’ve witnessed the spread of credential-itis and its pernicious side effects. Variations include demands that one defer to the master’s degree holder with no experience to engineers ignoring advice from the front line lead talking about the problems they’ve been watching develop all day. The opposite of this is the use of quality circles and open solicitation of ideas from those on the shop floor to improve operations.

 

“It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so.” The assumptions one makes that are incorrect cause the greatest problems. Peter Drucker’s theory of business called for challenging the assumptions on which the business model is based. Assuming the underlying assumptions are still true leads to mistakes, just as failing to see changes in the market and supply chain cause major failures down the line.

 

“Civilization is the limitless multiplication of unnecessary necessities.” One version of this in real life is the proliferation of features, each seen as essential because it was in the prior version.

Another variation is the continued collection of data, merely adding to the types of data collected. I’ve observed data collection performed in the hope that the data will be useful one day as well as suggested eliminating data collection fields as both labor savings and an error reduction technique.

The NSA took this to an illogical extreme, such that they were drowning in data in such volume they couldn’t find the information they needed that was the justification of the entire data collection effort in the first place. Streamlining data collection not only improves everyone’s efficiency but slows the growth of data repositories growing at 10%+ per year. In the case of online forms to be filled out by customers, cutting back on all the “necessary” data entry reduces shopping cart abandonment rates, as well.