A Summary of MSA Approaches for Attribute Measurement Systems

Attribute measurement systems (abbreviated as AMS in the rest of this article) are systems which differentiate the measured parts into some categories, such as OK and NG, instead of characterizing them with variable data. For the convenience of discussion, this article will further divide AMS into two types:

Type 1: AMS for characteristics which can also be evaluated with variable measurement systems. An example of such AMS is the system with a pin gauge. It differentiates the measured parts into two categories: OK or NG, but the diameter of the hole can also be measured with a variable measurement system such as CMM.

Type 2: AMS for true attribute characteristics which cannot be evaluated with variable measurement systems (abbreviated as VMS in the rest of this article). An example of such AMS is the manual visual inspection system on products appearance. The spec of appearance is often not defined with variable data.

In AIAG’s MSA manual (4^th edition), four approaches are introduced for the MSA of AMS, as listed below:

1.       Gauge performing curve

2.       Signal detection theory

3.       Attribute control charts

4.       Hypothesis test analysis

None of the above approaches can assess all the five types of variability of a measurement system, i.e. bias, linearity, stability, repeatability and reproducibility. They all have certain limitation as explained below:

The 1^st approached is an analytical method which gives good quantitative assessment on the measurement system. But it is only good to study the bias and repeatability. Also, it is only good for Type 1 AMS, as the reference value of the samples used in this study must be available in quantitative data. Readers may refer to AIAG’s MSA manual (Chapter III, Section C) for how to establish the gauge performing curve a measurement system. It is quite clearly illustrated there.

Same as the 1^st approach, the 2^nd approach can only be used for Type 1 AMS too. It can be used to assess the GRR and GRR only of the measurement system. It also gives a quantitative result, but it is an estimated one. The estimation becomes better with the increase of sample size. As this approach gives only an estimation of GRR, it should be approved by the customers before it is used.  

         The 3^rd approach is only briefly mentioned in the MSA manual (P.145, 4^th edition). This approach is only good for stability study. Same as the stability analysis of a VMS, one needs to follow the below steps: pick a sample, establish the control charts and then study the control charts (please refer to this article for stability study of VMS). Picking sample for stability study of AMS is much more critical than the stability study of VMS. It is probably the key for a good stability study of AMS. In AIAG’s manual (please refer to the footnote in P.145, 4^th edition), it’s noted that for the chosen sample, np must be greater than 4, i.e. if the sample is measured ten times, there are at least 4 times this sample is categorized as NG. So the sample should not be an OK sample which is far away from the spec limit, otherwise there is a low chance of getting an NG result from measurement. It should not be an NG sample which is far away from the spec limit either, otherwise, UCL and LCL are same and the control charts makes no sense (please refer to formula of UCL and LCL of p charts and np charts in the SPC manual). So the sample must be close to the spec limit in the gray area of the below image, with a chance of having both OK and NG results.  

In principle, this approach can be used for both Type 1 and Type 2 AMS, but if AMS involves human judgement, this approach may not work well, because the appraisers can remember the past judgements and it can influence the upcoming judgements. So this approach is good if the equipment makes the judgement, but not suggested if the appraisers make the judgement.  

The 4^th approach can be used for both Type 1 and Type 2 AMS. But unfortunately, the kappa value obtained with this approach can only assess the agreement between different appraisers and does not yield any quantitative assessment of any type of variation of the measurement systems. One cannot really tell how big the variation of the measurement system is with this approach. Therefore it should approved by the customers if an organization plans to use it. 

As a summary, below is the table which lists up the possible MSA approaches for AMS. Please be noted that the agreement analysis in the last column does not belong to the five types of MSA defined in the MSA manual.

How to Do Stability Study on a Measurement System

    Measurement system analysis includes five types of studies: bias, linearity, stability, repeatability and reproducibility. Among all of them, stability study is relevant easy and requires not too much concept in statistics. One can do it with some basic knowledge in SPC.

    Now let’s take a look at the detailed steps for stability study:

1. Pick a sample

    The first step is to pick a sample which will be measured and monitored. The sample is not necessary to be a calibrated standard with known value. It can just be any sample, and its value can also be unknown. But of course, same as any other analysis, this sample must have little within sample variation.

2. Collect the measurement data and establish the control chart with control limits

    Control charts, with established control limits, are needed for stability study. This can be done same as the conventional SPC (You may refer to this article about how to establish the control limits). The difference is, for process control, multiple samples are collected and one datum is read from each sample to form one subgroup, while for stability study, multiple readings are obtained from the same sample, as picked in Step 1, to form one subgroup.

    So how many readings should be obtained for each subgroup and how many subgroups should be collected before the control limits can be established? In P.86 of the 4th edition of AIAG’s MSA manual, 5 measurements are done in each shift, and totally 100 measurements are done to establish the control limits, i.e. 20 subgroups with a subgroup size of 5. But the 2nd edition of AIAG’s SPC manual suggests collecting at least 25 subgroups with at least 100 data points before establishing the control limits (P.57). My personal opinion is to stick with the rules in the SPC manual.

    An example of the control charts established is shown in the graph below. AIAG's MSA manual suggests using Xbar-R or Xbar-s control charts.

Figure 1 Control charts with control limits for stability study, cited from 4th edition of AIAG's MSA manual.

3. Analyze the control charts and determine the stability of the measurement system

    After establishing the control charts and control limits, one can then analyze the control charts and see whether any data point indicates a special cause. Same criteria for identifying the special cause in SPC, as shown below, can be used here.

Figure 2 Criteria for Identifying Special Cause, Cited from 2nd Edition of AIAG's SPC manual

    If no data point indicates a special cause in the control charts, then one can concludes that the stability of the measurement system is good. Otherwise, special cause exists in the measurement system and one needs to identify it and take actions to remove it to improve the stability.

4. Keep on monitoring 

    The stability study of the measurement system is completed up to Step 3. But if you’re careful, you don’t have to stop here. You can just keep collecting readings from the picked sample, with the defined subgroup size and sampling frequency, and plot the measured results in the control charts. By doing so, you are continuously monitoring the stability of the measurement system and actions can be taken whenever a special cause is detected. A continuous monitoring on the measurement system is certainly better than once a year study.

How to Calculate GRR%

    For GRR analysis on a measurement system, GRR% is used to determine whether the repeatability and reproducibility of the measurement system is acceptable for the purpose of measurement. It is calculated with Equation 1:

    In the AIAG’s MSA manual (4th edition), four different approaches are presented to determine TV based on different scenarios:

Accordingly, TV is calculated with the following equations:

    The above four approaches and the corresponding equations for TV may be confusing to many people who are in charge of MSA in their companies, and they may not be really clear of which equation should be used in actual cases they encounter when doing their GRR analysis. This article is intended to give some additional discussion and clarification about how to decide which equation should be used for TV and in turn GRR% calculation.

    To decide which equation to be used, one first needs to consider the purpose of measurement. In general, any measurement conducted on a process has two purposes:

1. To judge whether the measured parameter is within or out of the spec limit, for the purpose of OK/NG judgement or product control;
2. To decide whether the process is under control (without special cause) and whether the process capability (Cp and Cpk) and performance (Pp and Ppk) meet the requirements, i.e. for the purpose of statistical process control (SPC). 

1. When the measurement is for product control 

    When the measurement is for product control, one cares about the possibility of misjudgment. Misjudgment happens when the true value of a measured parameter is located near the spec limit, indicated with the gray region in the graph below.

The width of the gray region is determined by the variation resulted from the measurement system, i.e. the GRR of the measurement system (The width of gray region is 6 times of GRR. Outside this width, the possibility of misjudgment due to GRR is nearly 0). The wider the gray region is in respective of the tolerance (USL-LSL), the higher the possibility of misjudgment is. Therefore, in order to have a small possibility of misjudgment so that the measurement can really achieve the purpose of product control, the ratio between gray region and the tolerance 6*GRR/(USL-LSL) must be small enough. GRR% should hence be evaluated with the following equation, i.e. the 4th approach presented in the MSA manual: 

2. When the measurement is for process control

    When the measurement is for process control, one cares whether the obtained data can correctly identify the existence of special causes and whether the measurement can really reveal the true process capability (Cp and Cpk) and performance (Pp and Ppk). Let’s take a look at these two purposes respectively.

2-1. When the measurement is to identify the existence of special causes

    In AIAG’s SPC manual, it listed out 8 criteria to identify the existence of special causes, as shown below:

Among all the 8 criteria, the 1st one is mandatory, and the rest are optional. So the discussion here will be just based on the 1st criterion. 

    To identify the special causes with the 1st criterion, one needs to monitor whether there’s any data point out of the control limits (±3σp). It is similar to judge NG samples from good ones. The difference is that the former uses the control limits, while the latter uses the spec limits. To correctly identify the existence of special causes, one cares about the possibility of misjudgment (i.e. an under-control data point is misjudged as an out-of-control data point, or an out-of-control data point is misjudged as an under-control data point). This is determined by the ratio of the gray region around the control limits with respective to the width of the control limits, i.e. 6*GRR/(UCL-LCL). Therefore, GRR% should be evaluated with the following equation:

where σp is the process variation.

2-2. When measurement is to monitor the process capability or process performance

    For this purpose, let’s just see how GRR of a measurement system impacts the observed Ppk of a process. It’s similar for other index, including Cpk, Cp and Pp. 

    Ppk of a process is calculated with the following equation (please refer to P.133 of the 2nd edition of AIAG’s SPC manual)

where σp is the total process variation as observed from the measurement results. This observed process variation includes two sources: one is the actual variation of the process itself σ1, and the other is the variation from the measurement system, which is GRR. According to the property of the normal distribution

So Equation 8 can be rewritten as

It should be noted that Ppk as calculated above is the observed Ppk by the measuring process. It does not reflect the actual performance of the process. The actual Ppk should be calculated as below, without the variation from the measurement system in the denominator:  

Comparing Equation 10 with Equation 11, one can see that the larger GRR is, the bigger the difference is between the observed Ppk and the actual Ppk (the former is always smaller). In order to make process control effective, the observed Ppk must be as close as to the actual Ppk, so it causes no misjudgment on the actual process performance. In other words, GRR must be significantly smaller compared with σp, so that Equation 11 can be approximated with Equation 8. Therefore GRR% of the measurement system should be evaluated with the following equation:

This equation is equivalent to Equation 7. So for either purpose 2-1 or 2-2, it comes to the same conclusion, i.e. GRR% should be evaluated as the ratio of GRR with respect to the process variation.

    The process variation can be known or unknown when GRR analysis is conducted. Let’s now see how to calculate GRR% based on Equation 7 or 12 in both cases. 

    When the process variation is unknown, for example, GRR analysis is done when the process is still in design and development stage or the process has just undergone some changes and it has not been studied yet. In such a case, the observed process variation σp can be obtained with following approaches:

1. Select a group of samples from the process (It must be noted that the number of parts must be enough to cover the full width of the process variation) and measure the parts variation PV in this group. PV represents the true variation of the process, so the observed process variation σp, according Equation 9, is: 
   
   GRR%, hence, can be calculated with the following equation, i.e. the 1st approach introduced in MSA’s manual:

   
2. Use the process variation σp of a similar process, i.e. the 2nd approach:
   
   This approach should only be used when it’s not possible to obtain and do measurement on a group of samples representing the full process variation, i.e. the 1st approach is not feasible.
3. Use the target Pp or Ppk, i.e. the 3rd approach:

   

   Again, this approach should be used only if the 1st approach is not feasible.

    When GRR analysis is done when the process has already been studied and released to mass production, for example, the annual MSA after process release, the process variation is either directly known or can be derived from the other process index, and GRR% can be calculated with the following approaches:

1. The process variation is directly known, GRR% can be calculated with the 2nd approach.
2. When process variation is not known, but Pp is known, GRR% can be calculated with the 3rd approach. 
3. When the process variation is not known, but the variation of Xbar, i.e. or control limits of Xbar, i.e.is known, GRR% can be calculated with the following equation:
   
   This approach is not mentioned in the MSA manual
4. If a group of samples is available which represents the full width of the process variation, GRR% can also be calculated with the 1st approach, without using the existing data about the process. 

    As a summary, the table below lists up the scenarios what equations should be used for GRR% calculation：

做GRR分析时是否可以使用不合格品

      在网上看到有个问题：做GRR分析时，是否可以使用不合格品？这个问题的答案是不可以。以下是对于此答案的解释：
   
      进行GRR分析时，GRR%的计算可以根据不同情况采用三种公式（关于三种公式的解释，请参考另一篇文章）。其中，如果测量的目的是为了进行SPC管控，在尚未建立过程的控制限的情况下，GRR%的计算公式如下

此处整个分母代表的是用测量系统观察到的过程的总变化，包括测量系统引起的变化AV&EV和过程本身的变化PV（PV虽然被称为parts variation，但它由过程本身的能力决定，所以是过程本身变化的体现）。因为测量的目的是为了进行SPC管控，所以GRR分析时，需要评价的是测量系统的变化相对于受控状态下过程总变化的大小。如果进行GRR分析的样品中存在不合格样品时，PV将不再代表这个过程在受控状态下的变化（因为存在不合格，代表过程存在特殊因素，所以过程处于非受控状态），而是会大于处于受控状态的过程的变化。根据Equation I，因为PV变大，此时的GRR%将会比没有不合格品时候的GRR%来的小，造成对GRR分析结果的误判。

      另外，GRR%计算公式中的AV的计算公式如下

其中Xdiff是进行GRR分析的样品的平均值Xbar的最大值和最小值之间的差值。当存在不合时，这个差值会比没有不合格品的时候要大，从而影响AV的值，并进一步影响GRR%的值。

      综上所述，GRR分析时，不应该选择使用不合格品。

How to Decide What Analysis can be Exempted from MSA?

The AIAG's MSA manual requires that five types of analysis shall be done for MSA: bias, repeatability, reproducibility, linearity and stability. An MSA plan should cover all these five types for each measurement system appeared in the control plan as discussed in a previous article, unless it can be justified that certain analysis are not necessary. This article will discuss a couple of examples and explain how one decides what kind of analysis can be exempted from MSA.

Example I:

Caliper I is used for thickness check of materials A with a spec of 5+/-0.01mm. Caliper II is used for thickness check of material B with the spec of B 5+/-1mm. They are not used for any other measurements. In this case, Caliper I is used over a range around 4.99~5.01mm. Over such a small range, the impact of linearity to the measurement result is insignificant. On the other hand, Caliper II is used over a range around 4~6mm and the linearity of the measurement system could be a concern. Therefore study of linearity is not needed for measurement system of Caliper I, but is necessary for measurement system of Caliper II. 

Example II:

An electrical platform scale is used to measure the weight of each product before outgoing. There are two appraisers assigned for this measurement, but for an electrical platform scale, the appraisers should have no impact to the measurement result (all the appraisers do is just to turn on the machine and put the product on the scale. How they do these things do not change the measurement results), and hence in this case study of reproducibility is not necessary.

As a general rule, before deciding what kind of analysis can be exempted from MSA, one must first understand what variation the analysis evaluates. The table below is a summary:

Analysis	Variation Evaluated
Bias	Variation caused by all the factors of the measurement system
Repeatability	Variation caused by equipment itself
Reproducibility	Variation caused by different appraisers
Linearity	Variation caused by measurement range
Stability	Variation caused by time difference

When deciding any of the above analysis can be exempted, one must consider whether the corresponding variation evaluated by this type of analysis has any concern to the measurement results. If not, then the analysis can be exempted from MSA.