Section 12: Evaluation of milk analysers for ICAR Certification

Protocol for Evaluation of Milk Analysers for Granting ICAR Certification

This edition of teh ICAR Guidelines SEction 12 aims to integrate the previous Sections 12 and 13, thereby giving account of new developments in laboratory applications for milk analysis and in and on-farm measurement systems as well as the development of a revised Section 11 of the ICAR guidelines:

• For the procedural aspects alignment is sought with the revised Section 11.
• It covers milk analyser evaluation in two different options:
  • verification of milk analyser systems that are claimed to produce test data suitable for use in official milk recording, either operated in a laboratory or on-farm;
  • validation of claims for milk measurement systems that produce test data for farm management purposes, other than those for official milk recording, either operated in a laboratory or on-farm. In case ICAR does not wish to enter into these type of validation activities, the concerned text parts are to be skipped from the draft;

• It is stressed that the starting point of each verification or validation process is the content of the request and/or the claim of the applicant. Based on the evaluation of a claim by an ICAR Expert Panel, a standard test plan may apply or a dedicated test plan is to be developed. The test plan requires approval from ICAR and needs to be conducted by an ICAR appointed Test Centre. The test results will be evaluated by the ICAR Expert Panel, which will give a recommendation on ICAR verification (after a verification process) or ICAR validation (after a validation process).

• The text is considerably shortened and  made better accessible for the users by referencing to existing ISO|IDF standards and other relevant standards where possible.
• This document has been prepared by the ICAR Milk Analyses Sub-Committee, in close collaboration with the Chair and members of the ICAR Milk Recording and Sampling Devices Sub-Committee. ICAR expresses its deep appreciation to the author, Dr. Harrie van den Bijgaart, for his commitment and significant contribution to the development of this ICAR Guideline.

General Overview for Milk Analyser Evaluation

ICAR distinguishes between data use in the frame of official milk recording and for other farm management purposes. Section 12 of the ICAR Guidelines lists principles and procedures developed for verification of milk analyser systems against ICAR requirements in the frame of official milk recording and for validation of submitted claims with milk analyser systems for other purposes.

ICAR verification of milk analyser systems means performance evaluation according to stated testing and verification protocols for milk analysers for official milk recording purposes by an ICAR appointed Test Centre. In case of a successful verification, it has been demonstrated that the milk analyser system can deliver data as described in the test report and it meets the ICAR requirements as listed in Table 4, provided that the analyser is applied and operated according to the instructions of the manufacturer and with proper quality assurance in place when applied in routine.

ICAR certification is the final stage of the ICAR verification process for a milk analyser (system), resulting in a formal statement of ICAR on its suitability for use in official milk recording.

ICAR validation of claims with milk analyser systems means evaluation of performance with milk recording parameters or other parameters as claimed by an applicant, therefore following an ICAR approved test plan and being conducted by an ICAR appointed Test Centre. As the claims may vary widely, no standard test plan can be stated. A customised test plan will therefore be developed on a case-by-case basis, based on an clear-cut claim from the applicant. In case of a successful validation, it has been demonstrated that the quality of delivered data can comply with the stated claim of the applicant, provided that the system is applied and operated according to the instructions of the manufacturer and with proper quality assurance in place when applied in routine.

ICAR certificate of validation  is the final stage of the ICAR validation process for a milk analyser (system), resulting in a formal statement of ICAR on compliance with one or more performance claims as stated by the applicant.

After that the certificates of validation and/or verification are released, ICAR does require the applicant to prove that the concerned milk analyser system continues to comply with the applicable evaluation criteria during the conducted ICAR verification or ICAR validation process. This includes annual reporting by applicants and timely notification of ICAR with the introduction of modifications to a certified claim.

Laboratory analysers in use before 1 January 2016 that have been previously accepted by ICAR Member Organisations are exempted from the need to be ICAR-verified for the use in official milk recording.

An annex with this Section provides guidance for analytical quality assurance with the application of milk analysers in practice.

Figure 1. Flowchart for ICAR evaluation of milk analyser systems with maximum timeline, provided that all actors strictly adhere to the allocated time per step.

In case milks of different animal species are to be analysed, specific evaluations for every species concerned have to be carried out to assess that the instrument is appropriate for the intended use. Refer to Table 1 for species-specific component ranges.

In case of breeds with unusual milk fat and protein contents (i.e. Jersey breed with high fat and protein contents), the evaluation should cover the component range for milk of the specific breed.

In case of new devices or systems, a full test is carried out, while for modified devices or systems a partial test or a desk review may suffice, according to the decision of the ICAR Chief Executive based on the recommendation of the concerned ICAR Expert Panel.

Definitions

Reference standard

A reference standard is a measurement standard designated for providing reference values, in order to calibrate a quantitative alternative method. A reference standard can be:

1. certified or recognized reference material with assigned values, or
2. reference method. A reference method should be an internationally standardised method (i.e. ISO, IDF, AOAC methods, see Annex A with this Section), or
3. a rapid method as an alternative for a reference method. Such a rapid method can be used in case results have shown to be equivalent to those from the reference method (i.e. Gerber method for fat content, Amido Black method for protein content, enzymatic method for lactose content).

A master instrument may be used to produce reference values, as long as these are anchored to one of the abovementioned reference standards, for calibration of other instruments and for other operators in case of a system with centralised calibration. Application of a centralised calibration concept must take into account the sensitivity of routine methods to matrix effects (milk composition).

Alternative method

Method fit for purpose based on a performance evaluation, using an internationally standardised or otherwise approved protocol. Conditions and approved procedures for ICAR verification or validation, as well as requirements for ICAR certification, are defined in standard protocols.

Other definitions

Milk analyser (system): analytical device specifically dedicated to the analysis of milk. It may pertain to automated methods in laboratories as well as to devices for milk analysis installed on-farm. It may consist of a sole analytical device, but also be combined with a sampler and/or algorithms for data treatment in order to arrive at final test results.

Laboratory milk analyser (system): milk analyser (system) installed in a laboratory that is used for analysing a representative sample once the whole milking is obtained and used either to detect or to quantify one or more components or characteristics in milk.

.On-farm at-line milk analyser (system): milk analyser (system) installed beside a production line on a farm that is used for analysing a representative sample once the whole milking is obtained and used either to detect or to quantify one or more components or characteristics in milk. Such devices are likely to be located close to the milking unit but can also be placed elsewhere on the farm. They can have similar characteristics as those used in laboratories and be an element of an on-farm laboratory.

NOTE Also called off-line analysers

On-farm in-line milk analyser (system): milk analyser (system) installed in the milking system (i.e. milking equipment, milk pipeline and/or automatic milking system) on the farm that is used to detect or to quantify one or more components or characteristics in milk of one animal passing by during the milking process

Representative test result: a result either obtained:

• through collection of the whole milking and then, after proper mixing, withdrawal of a sample and subsequently measuring it

or

• by serial in-line measurements of milk in weighted proportion to the total quantity of milk produced and calculation as a weighted average for the whole milking.

Repeatability: limit value for the absolute difference between two final values, each of them representing a series of individual test results or measurement results obtained with the same method on identical test measurement items in the same analyser system by the same operator using the same analyser system within short intervals of time, is expected to be with a specified probability of 95 %.

Intra-reproducibility: limit value for the absolute difference between two final values, each of them representing a series of individual test results or measurement results obtained with the same method on identical test measurement items in the same analyser system by the same or another operator using the same analyser system within short to intermediate intervals of time, is expected to be with a specified probability of 95 %.

Carry-over: residual milk included in the present milk sample from the previously tested milk sample, expressed as a percentage of the mass of the present milk sample.

Linearity: Constancy of the ratio between the increase in the concentration of a milk component or value of a milk characteristic and the corresponding increase of the measurement method result. It is expressed as the ratio of the residual range to the signal values range.

Accuracy (or overall accuracy): closeness of agreement between a measured quantity value (or estimate) obtained with an analytical method and a true quantity value of a measurand. It is expressed through a standard deviation that combines both random error (precision) and systematic error of the method. The part independent from calibration and precision errors, so-called ‘accuracy of estimate’, is a characteristic of alternative methods of analysis.

Bias (or systematic error): the difference between an average of test results (over time) obtained with the alternative method and the average of test results obtained by reference method testing on the same test items.

Measurement uncertainty: related to overall accuracy of the method, it expresses the range of occurrence of a result through its standard deviation (standard uncertainty) and a coverage factor k for a given probability (usually k=2 for a 95% probability). It is presumed that the resulting error is normally distributed.

Sensitivity: the ability of a qualitative method to detect the analyte compared to the reference. True positive rate = # true positives/(# true positives + # false negatives).

Specificity: the ability of a qualitative method not to detect the analyte when it is not detected by the reference. True negative rate = # true negatives/(# true negatives + # false positives).

Procedure with Application for ICAR Verification of Milk Analysers and ICAR Validation of Claims with Milk Analysers

The procedure for any type of verification or validation starts with a request or claim by an applicant, which can be a manufacturer, another stakeholder or a combination of these, to the ICAR Secretariat. The applicant should apply through the application form available through the ICAR Secretariat and ICAR website. A complete application includes:

• General features of the applicant (name, addresses, physical location, contact person), to be provided filling an ICAR application form;
• Information on outsourced processes used by the applicant that are relevant for the evaluation;
• Milk analyser name and type, additional ‘brand and type’ names;
• Purpose and measurement parameters;
• Where and how to be applied (laboratory, on-farm at-line, on-farm in-line);
• Mounting position or use;
• Specie(s) - i.e. cow, buffalo, sheep, goat, other;
• Peer reviewed publications;
• Reports of validation studies;
• Claim(s) to be verified (official milk recording) or validated (other);
• Test type required (new device, testing a modification);
• Technical characteristics, drawings and photographs of device;
• Technical manual outlining functional processes and principles as well as software/firmware documentation;
• Installation procedure;
• User manual, including instructions on proper maintenance;
• Routine test or periodic check procedures for operators and service technicians.

Following the payment of a first part of the application fee by the applicant, the following steps will be taken:

• ICAR Secretariat will check the application form and related documentation.
• ICAR will recruit and appoint experts to form an ICAR Expert Panel. This Expert Panel reviews the application with the request/claim.
• ICAR, on recommendation of the ICAR Expert Panel will appoint an ICAR Test Centre to conduct the tests.
• The ICAR Expert Panel will formulate a test plan in consultation with the applicant and the ICAR appointed Test Centre.
• ICAR issues an umbrella contract and sends it to the applicant together with the test plan and the invoice for the full test fees.
• The ICAR test will be scheduled and conducted after the contract is signed and the test fees are paid in full. The ICAR Secretariat coordinates financial transactions between the applicant, the ICAR appointed Test Centre, and ICAR.
• For the test to begin, the applicant must send all the necessary devices and accessories to the ICAR appointed Test Centre. In addition, all manuals, documents, and procedures should be provided to the ICAR appointed Test Centre.
• Then the ICAR appointed Test Centre will conduct the tests. The ICAR appointed Test Centre is obliged to act according to the procedures laid down in the test plan and accompanying protocols. All details associated with the testing phase, including the test results, are kept strictly confidential. The ICAR appointed Test Centre provides periodic reports on tests in progress to the concerned ICAR Expert Panel.
• Upon completion of the test, the ICAR appointed Test Centre provides a draft report on the test results to the concerned ICAR Expert Panel through the ICAR Secretariat.
• After a 21-day review and comment period by the ICAR Expert Panel, the ICAR appointed Test Centre addresses any possible comments, prepares a final report and resubmits it to the ICAR Expert Panel through the ICAR Secretariat within 21 days of the receipt of the comments.
• After another 21-day review period, the ICAR Expert Panel provides its recommendation on validation or verification output to ICAR Certification with a notification to the applicant.
• In case of a successful test for verification, ICAR Certification will grant an ICAR certificate of verification  to the applicant. In case of a successful test for validation, ICAR will grant an ICAR certificate of validation to the applicant.
• All ICAR-verified milk analysers (systems) and ICAR-validated claims are published on the ICAR website, including pictures of the device, year of verification or validation, species and information on the certified claim.

Verification of Milk Analyser Systems (against criteria for use in official milk recording)

The manufacturer or applicant is responsible for the correct installation and calibration of the milk analyser or milk analyser system. After installation, the ICAR appointed Test Centre will conduct the tests without representatives of the manufacturer or the applicant present.

Laboratory and on-farm at-line analyser systems

Scope and rationale

This protocol is applicable to the verification of alternative methods for the quantitative determination of milk parameters intended to generate data for official milk recording. Examples are fat, protein, true protein, lactose and urea content, and somatic cell count. Specific concentration ranges for these parameters in milk of different species are listed in Table 1. This protocol can in future be extended to other milk parameters, such as fatty acids. The protocol applies for laboratory methods as well as for on-farm at-line methods, which measure a single representative sample of a whole milking. On-farm at-line analyser systems allow more frequent milk analysis by farmers. Therefore, a lower accuracy at the level of the individual result can be accepted. In case that an at-line analyser system is used to replace the laboratory system at a usual recording frequency, for instance, where the sample transportation to laboratories is impractical, it should comply with the ICAR guidelines for laboratory analysers.

Table 1. Indicative milk component ranges at least to be covered by a verification.

	Cows	Goats	Sheep	Buffaloes	Units
Fat	2.0 – 6.0	2.0 –5.5	5.0 – 10.0	5.0 – 14.0	g/100 g
(True) Protein	2.5 – 4.5	2.5 – 5.0	4.0 – 7.0	4.0 – 7.0	g/100 g
Lactose	4.0 – 5.5	4.0 – 5.5	4.0 – 5.5	4.0 – 5.5	g/100 g
Urea	10.0 – 70.0	10.0 – 70.0	10.0 – 70.0	10.0 – 70.0	mg /100 g
Somatic cells	0 – 2 000	0 – 2 000	0 – 2 000	0 – 2 000	103 cells/ml

Recommendations for safeguarding milk sample quality

With laboratory analysis and on-farm at-line analysis, it is essential that each milk sample is a representative sample for the whole milking or the commingled milkings. Guidance for representative sampling is in Section 2, Chapter 7.1.

Milk samples are to be properly preserved until measurement. With prolonged storage and transport, chemical preservatives can be used according to guidance in ISO 9622|IDF 141 or ISO 13366-2|IDF 148- Since chemical preservatives can influence the performance of analytical methods, and if the method is not specified in the ISO standards referenced above, any specific effects of the preservative and its possible accompanying substances must be evaluated, documented, and considered before applying sample preservation in routine testing.

The impact of preservatives can be assessed by comparing measurement results from preserved and unpreserved portions of the same samples. Also, be aware of the (small) dilution effect when preservatives are added as a liquid.  

Even with the use of chemical preservatives, sample are preferably to be kept cold (0-6 °C) during storage and transport. Storage at elevated temperature (>20 °C) during one or more days may for instance result in lower somatic cell counts.

Sample vials should be fit for use. Care shall be taken that sample bottles are leak-proof and that a proper empty volume is left. A too large empty volume can cause oiling-off; a too small empty volume can cause problems with mixing.

Performance characteristics to be examined

Range

A quantitative method must be able to provide accurate results over the range of interest (see Table 1). This means showing low enough variability (short term, medium term and between instrument variability) but also showing accurate results over the relevant concentration range. Deviations of the latter will become apparent when comparing alternative method results against reference method results or results from instruments calibrated and traceable to reference material as non-compliant variability of differences or non-linearity (curving).

Repeatability, medium term stability and intra-reproducibility

With any analytical method, a fair level of short-term signal and result stability is a primary requirement. Therefore, an examination of the repeatability is at the basis of any verification process. Medium-term stability (same analyser system during one day or over several days) and between-instrument variability are other key characteristics to be examined. These are expressed in terms of (intra-) reproducibility.

For testing repeatability and medium-term stability, a method or instrument should be operated in conditions as close as possible to routine. The repeatability and the medium-term stability are to be evaluated at three different concentrations of the measurand, low, medium and high. For an appropriate test protocol, guidance on the minimum number of samples to be tested per level and calculation of the standard deviation of repeatability and intra-reproducibility from the measurement results, see ISO 8196-3|IDF 128-3.

For determination of between-instrument variability or between-location variability, the same sample set is to be analysed with different analyser systems or different locations, thereby safeguarding that the composition and the condition of the samples do not differ. The methods and/or analyser systems should be operated in conditions as close as possible to routine. The between-instrument variability or between-location variability are to be evaluated at three different concentrations of the measurand, low, medium and high. Guidance on the test protocol, the minimum number of samples to be tested per level and the calculation of the standard deviation of between-instrument or between-location reproducibility is provided in ISO 8196|IDF 128-3.

Applicable ICAR limits for the standard deviation of repeatability, medium term repeatability and intra-reproducibility for official milk recording parameters are listed in Table 4.

Carry-over

Milk adhering to stirring devices or incomplete rinsing of the flow system and the measuring cell between samples may result in transfer of milk to a next sample when measured. Especially with a large difference in component contents between two successive milk samples analysed, this can be noticeable in the outcome of the measurement of the latter sample.  

In the evaluation of a method, this so-called carry-over effect is to be assessed as described in ISO 8196-3|IDF 128-3.

Applicable ICAR limits for the carry-over ratio with official milk recording parameters are listed in Table 4.

It is noted that some methods/analyser systems are coming with a software feature that allows to automatically correct for the overall carry-over effect. Automatic correction of milk test results is acceptable within limits, provided it can be proven that the transfer of a small quantity of material from one measurement to the next is systematic and stable. It is stressed that the limits in Table 4 apply for the situation without any correction, that means that with the assessment of the carry-over ratio the carry-over compensation is set to zero.

Linearity (If applicable and relevant)

Linearity expresses the constancy of the ratio between the increase in the component concentration in milk and the corresponding increase of the test result. Linearity of the measurement signal is in most cases essential to maintain a constant sensitivity over the measuring range and to allow easy handling of calibration and related settings. Moreover, it allows to some extent measurements beyond the calibration range through linear extrapolation.

In the evaluation of a method or analyser system for milk analysis, the linearity is to be assessed as described in ISO 8196-3|IDF 128-3.

Applicable ICAR limits for the linearity ratio with official milk recording parameters are listed in Table 4.

If linearity is not achieved throughout the whole concentration range, the actual range of application for the method concerned is to be determined or separate measurement ranges with acceptable linearity have to be indicated.

The lower limit of the measurement range is either the limit of quantification (test result with zero content + 10*standard deviation of repeatability) or the lower threshold where the signal of the measurement deviates from linearity, whatever is highest. It is noted that natural concentration ranges for the milk recording parameters are normally considerably above zero, whereby the lowest routine test result falls within the linearity range of the method.

The upper limit of the measurement range of a method corresponds to the upper threshold where the signal or the measurement deviates from linearity.

Accuracy

The overall accuracy error is the sum of repeatability error, the accuracy error (error of estimates versus reference) and the error of calibration which occurs in routine analytical conditions. For more general background information, see ISO 8196-1|IDF 128-1.

The overall accuracy is determined through the analysis of individual milk samples with both the alternative method and the reference method.

The evaluation is to be performed on the milk analyser system in the same state (working parameters, speed, calibration) the manufacturer intends to provide users with.

In case different analytical speeds are available, the overall accuracy is to be assessed for each of the options.

The overall accuracy is to be assessed as described in ISO 8196-3|IDF 128-3.  

With on-farm at-line analyser systems a somewhat lower performance as compared to laboratory instruments is acceptable with analysis that is more frequent, since measurement uncertainty can be reduced by averaging over multiple samplings

Applicable ICAR limits for the overall accuracy in individual animal samples with official milk recording parameters are listed in Table 4.

Ruggedness and user-friendliness

During the verification, the downtime, the frequency and type of required service operations are to be registered for reporting and evaluation by the ICAR Expert Panel. The alternative method should also be assessed for general convenience aspects such as safety in use, speed, consumables, environment-friendliness, user-friendliness (ease and ruggedness) in operation, cleaning, calibration, measures against unauthorized alteration of settings, quality assurance, performance monitoring, troubleshooting and servicing.

Overview of performance characteristics to be verified with laboratory and on-farm at-line analyser systems

The performance characteristics to be examined in a verification process will depend on the type of analyser system, where and how the analyser in applied, the measurement parameters and the content of the claim of the manufacturer. For verification with laboratory and on-farm at-line analyser systems an overview is provided in Table 2.

Table 2. Performance characteristics to be examined with verification of laboratory and on-farm at-line milk analyser systems.

	Fat	(True) protein	Lactose	Urea	Somatic cell count	Other
Range	Yes	Yes	Yes	Yes	Yes	Yes
Repeatability	Yes	Yes	Yes	Yes	Yes	Yes
Intra-reproducibility	Yes	Yes	Yes	Yes	Yes	Yes
Carry-over	Yes	Yes	Yes		Yes	If relevant
Linearity	Yes	Yes	Yes	Yes	Yes	If relevant
Accuracy	Yes	Yes	Yes	Yes	Yes	Yes
Mean bias	Yes	Yes	Yes	Yes	Yes	Yes
% outliers	Yes	Yes	Yes	Yes	Yes	Yes
Ruggedness and user-friendliness	Yes	Yes	Yes	Yes	Yes	Yes

On-farm in-line analyser systems

Scope and rationale

With on-farm in-line analyser systems, many approaches in obtaining representative test results for a milking or a series of milkings are possible. Method providers can address the challenge by making the optimal choices in combinations of the type of sampling device, the moments and the frequency of sampling, the type of measurement device, the number of measurements and the use of specific algorithms in data processing. ICAR chooses to leave degrees of freedom in continuous development and improvement with the method providers as much as possible. For performance evaluation of any in-line solution, ICAR focuses on the performance in a field test. Key parameters are the accuracy of the final test result of the whole analyser system for a single whole milking (or for the commingled milkings during a day) and the between-device variability of test results. Whereas data are intended to be used in official milk recording, these test results should meet with the demands for accuracy and intra-reproducibility for on-farm at-line analysers.

Furthermore, demands with regard to ruggedness and user-friendliness apply.

1.2.2  Performance characteristics to be examined

General

A milk analyser system shall be designed to permit easy reading and handling by the operator while it is attached to the milking equipment. In addition, it shall be resistant to all conditions encountered in its normal working environment (i.e. during milking, washing, disinfecting and, when applicable, transport). All parts subject to wear and tear shall be easily replaceable.

The conditions for assembling a milk analyser system are to be provided by the manufacturer of the device. If a milk analyser system is fitted with a calibration device or calibration option, adequate precautions shall be taken to prevent unauthorized alteration of settings.

A milk analyser system offered for ICAR verification on suitability for use in official milk recording shall be able to estimate the fat and (true) protein content of a milking or of the commingled milkings during a 24-hour period. Other parameters as lactose, urea and somatic cells are not obliged, but could be a part of the evaluation test on request of the applicant.

Range

An in-line milk analyser system must be able to provide accurate results over the range of interest, see Table 1. This means showing low enough variability (short term, longer term and between device variability) but also showing accurate results over the relevant concentration range. Deviations of the latter will become apparent when comparing alternative method results against reference method results or results from instruments calibrated and traceable to reference material, either as non-compliant variability of differences or non-linearity (curving).

Accuracy

For the evaluation of the total accuracy, a comparison is made between the estimate for the whole milk or commingled milkings over a 24-hour period with reference method measurements in duplicate on representative samples of these milkings. The estimates should come from a measurement system in the same state (moment and frequency of sampling, measurement device, speed, algorithms, calibration) the manufacturer intend to provide users with. The performance of the whole milk analyser system is to be compared against the accuracy limits stated for on-farm at-line analyser systems, thereby accounting for the fact that also here analysis for milk recording parameters is likely to be performed more frequently than in the classical system with laboratory analysis.

While collecting the milk from one whole milking or the milkings during a day, care should be taken that during the collection, the quality of the milk is preserved by working with clean utensils under sanitary conditions and with cold storage (0-6 °C) of the milk until the moment of sampling for reference method analysis. Guidance for representative sampling is in Section 2, Chapter 7.1.

Milk samples for reference method analyses are to be properly preserved until measurement takes place. With prolonged storage and transport, chemical preservatives can be used according to guidance in ISO 9622|IDF 141 or ISO 13366-22|IDF 148-2 Since chemical preservatives can influence the performance of analytical methods, and if the method is not specified in the ISO standards referenced above , any specific effects of the preservative and its possible accompanying substances must be evaluated, documented, and considered before applying sample preservation in routine testing. Even with the use of chemical preservatives, milk samples are to be kept cold (0-6 °C) during storage and transport.

Sample vials should be fit for use. Care shall be taken that sample bottles are leak-proof and that a proper empty volume is left. A too large empty volume can cause oiling-off; a too small empty volume can cause problems with mixing.

The overall accuracy is to be assessed as described in ISO 8196-3|IDF 128-3, whereby at least 100 data pairs should be collected per parameter, with 50 in the medium range and 25 in each of the low and the high measurement ranges according to Table 1.

As with on-farm at-line analyser systems, a somewhat lower performance as compared to laboratory instruments is acceptable with analysis that is more frequent since measurement uncertainty can be reduced by averaging over multiple samplings. However, situations with a systematic bias should be avoided. Therefore, applicable ICAR limits for the overall accuracy in individual animal samples with official milk recording parameters at different levels in Table 4 are listed in conjunction with limits for the maximum bias and a maximum number of outliers.

Note: it is advised that in routine measurement results are reported together with results of pilot checks, listing both the assigned value and the measured values for the pilot sample(s).

Between-device reproducibility

Milking the same milk cannot be performed twice per animal. However, assessment of between-device variability may be feasible by mounting two analyser systems in the same milking device/milking line, resulting in two estimates for the milking or commingled milkings in parallel. For a proper indication of between-device reproducibility, this has to be replicated for at least 40 milkings.

NOTE     This test can be combined with the evaluation of total accuracy.

Guidance on the determination and the calculation of the standard deviation of between-device reproducibility is provided in ISO 8196|IDF 128-3.

Applicable ICAR limits for the standard deviation of between-device reproducibility for official milk recording parameters are listed in Table 4.

Ruggedness and user-friendliness

During the verification, the downtime, the frequency and type of required service operations are to be registered for reporting and evaluation by the ICAR Expert Panel. The alternative method should also be assessed for general convenience aspects such as safety in use, speed, consumables, environment-friendliness, user-friendliness (ease and ruggedness) in operation, cleaning, calibration, quality assurance, performance monitoring, troubleshooting and servicing.

Overview of performance characteristics to be verified with on-farm in-line analyser systems

The performance characteristics to be examined in a verification process will depend on the type of analyser system, where and how the analyser in applied, the measurement parameters and the content of the claim of the manufacturer. For verification with on-farm in-line analyser systems an overview is provided in Table 3.

Table 3. Performance characteristics to be examined with verification of on-farm in-line milk analyser systems.

	Fat	(True) protein	Lactose	Urea	Somatic cell count	Other
Range	Yes	Yes	If relevant	If relevant	Yes	If applicable
Repeatability	if applicable	If applicable	If relevant and applicable	If relevant and applicable	If feasible	If relevant and applicable
Between-device reproducibility	Yes	Yes	If relevant	If relevant	Yes	If relevant
Carry-over	If applicable	If applicable	If relevant and applicable	If applicable	If applicable	If applicable
Linearity	If applicable	If applicable	If relevant and applicable	If applicable	If applicable	If applicable
Total accuracy	Yes	Yes	If relevant	If relevant	Yes	If relevant
Mean bias	Yes	Yes	If relevant	If relevant	Yes	If relevant
% outliers	Yes	Yes	If relevant	If relevant	Yes	If relevant
Ruggedness and user-friendliness	Yes	Yes	If relevant	If relevant	Yes	If relevant

Table 4. Limits for test bed evaluations of milk analysers for use in official milk recording.

Component		Fat	(True) Protein	Lactose	Urea	Somatic cell count
Units		g/100 g	g/100 g	g/100 g	mg/100 g	103 cells/ml
Number of animals		100	100	100	100	100
Range	Total range			4.0-5.5	10.0-70.0	0-2 000
	Low					0-100
	Medium	2.0-6.0	2.5-4.5			100-1000
	High	5.0-14.0	4.0-7.0			>1 000

Parameter	Laboratory (At-line / In-line)			On-farm (Whole milking)
	Fat / True Protein / Lactose (g/100 g)	Urea (mg/100 g)	Somatic Cell Count (%)	Fat / True Protein / Lactose (g/100 g)	Urea (mg/100 g)	Somatic Cell Count (%)3
Repeatability standard deviation sr – Total range		4			10
Repeatability standard deviation sr – Low		6			15
Repeatability standard deviation sr – Medium	0.014	1.4	4	0.035	3.5	10
Repeatability standard deviation sr – High	0.014		3	0.035		7.5
Intra reproducibility standard deviation sRintra¹ – N.	40	40	40	40	40	40
Intra reproducibility standard deviation sRintra¹ – Total range		5			12.5
Intra reproducibility standard deviation sRintra¹ – Low		7			17.5
Intra reproducibility standard deviation sRintra¹ – Medium	0.020	2.0	5	0.05	5	12.5
Intra reproducibility standard deviation sRintra¹ – High	0.025		4	0.06		10
Carry-over limit Lc (%) – Total range	1		2
Linearity ratio De/DL (%) – Total range	1	2	2
Accuracy standard deviation sy,x² – Total range	0.06	6.0	10	0.15	15	25
Mean bias – Total range	±5.000	±1.2	±5	0.125	±3	±12.000
RMSEP – Total range				0.15	15
RMSEP% – Total range						25
Maximum number of outliers (%) – Total range	3	3	3	3	3	3

¹⁾Applies for medium-term, between-instrument, between-device and between-location reproducibility

²⁾Accuracy standard deviation s_y,x means residuals standard deviation  and not bias standard deviation

³⁾If not differently specified

Validation of Claims with Milk Analysers

Scope and rationale

Apart from official milk recording, milk analysis may in many other ways contribute to informed farm management decisions. Data on milk composition or other characteristics may be used as such, or in time series, in combination with milk data from other animals in the herd or other type of data for obtaining indications on production, animal health and welfare, as well as sustainability aspects. The increase in the application of mathematical models and algorithms in offered solutions is well noted.

Due to the wide variation in traits, purposes and solutions, it is not feasible nor constructive to develop an all-encompassing protocol for the evaluation of such applications. Still, it is important that users can have trust in the functionality of offered solutions. In order to support this but leaving degrees of freedom in development, ICAR offers to validate claims with solutions that are intended for use other than for official milk recording according to the procedure in clause 4.

Recommendations for safeguarding milk sample quality

The same recommendations apply as with the verification of on-farm in-line milk analyser systems, see clause 4.1.2.

Performance characteristics to be examined

Depending on the claim, the validation plan may address:

• For qualitative methods: carry-over, sensitivity, specificity, robustness, user-friendliness.
• For quantitative methods: range, zero/blanks, repeatability, within-lab or between-device reproducibility, carry-over, accuracy, robustness, user-friendliness.

Certification for Validation or Verification

The certification process follows the ICAR Guidelines Section 11 process.

To be noted that verification (sect.12) = certification section 11

Granting of Certificates

ICAR certification is the final stage of the ICAR testing and certification process for milk analyser systems.

A positive outcome of an ICAR verification study will be presented with an ICAR certificate stating the suitability of the evaluated milk analyser system for use in official milk recording.

A positive outcome of an ICAR validation study will be presented with an ICAR certificate stating the justifiability of the stated claim with the evaluated milk analyser system.

This under the provision that an analyser system is operated with strictly following the appropriate manufacturer’s instructions for use and the applicable guidelines for analytical quality assurance.

The certificate or attest will indicate the field of application for which the analyser has been evaluated (animal species, component/attribute, concentration ranges, etc.)

Conditions on the Use of ICAR Certificates

ICAR certification requires the applicant to comply with a number of conditions.

Certification include ongoing surveillance on the use of ICAR Certificates in the public domain and in communications of the applicant and others, of the manufacturing process and the functioning of the product. ICAR can perform audits with the applicant to verify whether the conditions for granting and/or keeping an ICAR certificate or attest are met. For that, the applicant should give access to premises, employees, documents and data carriers as requested by ICAR and concerned auditors appointed by ICAR. In the frame of an audit, ICAR will respect confidentiality aspects.

Modifications of Certified Milk Analyser Systems

In case a certified milk analyser system is modified in hardware, software and/or operating instructions, possibly influencing the outcome of measurements or having an effect on the justifiability of an ICAR certified claim, the applicant is responsible to report the modification(s) to ICAR. ICAR will consult experts and present to the applicant the required plan of action. If applicable, the applicant requests to review the device or method modification to ICAR using the normal test application process. The retest, which may include a desk review, modification test or a full test, will be contractually managed by ICAR as in the case with testing a new milk analyser system or claim with a milk analyser system.

Annual Reporting

On an annual basis, ICAR Secretariat contacts the applicant of a certified device or attested claim to confirm which of the concerned milk analyser systems are still sold and/or operated in various countries, and to report any modifications since the previous year’s report.

The applicant is responsible to provide the following information:

1. Names and models of the ICAR-certified milk analyser system or description of an ICAR-certified claim;
2. Modifications, if any, made on any or all ICAR-certified milk analyser systems or changes in claims;
3. Other companies with right to use/manufacture the concerned certified milk analyser systems or are using a claim, and under which name(s) these are represented in the marketplace;
4. If (c.) applies, report any modifications made by the other company(ies) as reported in (c.);
5. List of countries where the milk analyser systems are marketed;

A template for the report by manufacturers of ICAR-certified devices and ICAR validated sensor systems is available from the ICAR Secretariat.

The applicant must sign the document and submit it to the ICAR Secretariat within the deadline communicated by ICAR. ICAR will arrange for review of the annual reports submitted by the applicants. In case of previously unreported modifications or sufficient evidence of problems with a device, ICAR will communicate with the applicant for response and action.

Expiration and Prolongation of Certificates

ICAR certificates are valid for a period of 5 years from the date of issue. The applicant may request prolongation thereby following the applicable ICAR procedures.

With a prolongation request ICAR will give careful consideration to the content of the annual reports, audit results, reported modifications and any other information shared by the applicant or obtained from other sources.

ICAR Secretariat will share its decision on prolongation within 3 months with the applicant and, if applicable, issue a prolongation certificate.

Suspension, Reduction in Scope and Withdrawal of Certificates

In case of sufficient evidence of non-conformance issues, ICAR will communicate with the applicant for response and corrective action. Failure to meet the conditions may lead to the suspension, reduction in the scope or withdrawal of the ICAR certificate.

References

International Dairy Federation. International Harmonized Protocol for Proficiency Testing of (Chemical) Analytical Laboratories. Bulletin of the IDF 342:1999.

ISO 8196-1:2009|IDF 128-1:2009 – Milk - Definition and evaluation of the overall accuracy of alternative methods of milk analysis - Part 1: Analytical attributes of alternative methods. International Organization for Standardization/International Dairy Federation. 2009.

ISO 8196-2:2009|IDF 128-2:2009 – Milk - Definition and evaluation of the overall accuracy of alternative methods of milk analysis - Part 2: Calibration and quality control in the dairy laboratory. International Organization for Standardization/International Dairy Federation. 2009.

ISO 8196-3:2022|IDF 128-3:2022 – Milk - Definition and evaluation of the overall accuracy of alternative methods of milk analysis - Part 3: Protocol for the evaluation and validation of alternative quantitative methods of milk analysis. International Organization for Standardization/International Dairy Federation. 2022.

ISO 9622:2013|IDF 141:2013 – Milk and liquid milk products – Guidelines for the application of mid-infrared spectrometry. International Organization for Standardization/International Dairy Federation. 2013.

ISO 13366-2:2006|IDF 148-2:2006 – Milk– Enumeration of somatic cells - Part 2: Guidance on the operation of fluoro-opto-electronic counters. International Organization for Standardization/International Dairy Federation. 2006.

ISO 13528:2022 - Statistical methods for use in proficiency testing by interlaboratory comparison. International Organization for Standardization. 2023.

ISO 17034:2016 - General requirements for the competence of reference material producers. International Organization for Standardization. 2016.

ISO/IEC 17043:2023 - Conformity assessment - General requirements for the competence of proficiency testing providers. International Organization for Standardization. 2023.

ISO Guide 33:2015 - Reference materials – Good practice in using reference materials. International Organization for Standardization. 2015.

Annex A: Analytical Quality Assurance

Reference methods, reference materials

For the overview of reference methods, see Annex B.

Reference materials used for DHI analytical purposes are to be produced in quality assurance conditions, according to international standards, or failing that, international guidelines or agreements:

1. ISO 17034
2. ISO Guide 33403. 

Choice of Analytical Quality Assurance (AQA) service suppliers - i.e. for reference materials and proficiency testing - is to be made carefully. Services suppliers should operate under quality assurance and be able to provide documented proof of that. Service suppliers should submit themselves to a periodical independent audit, i.e. a third party, in order to have the conformity of their QA system judged. These audits can be carried out by accreditation assessors, commissions of user representatives, experts acting on behalf of the national DHI organisation, provided that their competence and independence are guaranteed and that the audits are conducted in line with ISO and ILAC recommendations.

Quality control in routine milk analysis

Performance of testing methods has to be monitored. For more background information, reference is made to ISO 8196-2|IDF 128-2.

Internal control

Irrespectively of the parameter, internal quality control on analytical methods is prerequisite for valid results. Specific standards have to be applied where they exist:

1. Fat, (true) protein, lactose and urea (mid infra-red spectrometry): ISO 9622 | IDF 141;
2. Somatic cell count: ISO 13366-2 | IDF 148-2.

According to ISO 8196 | IDF 128, the major checks in quality control are on:

1. Repeatability;
2. Daily and short-term stability of instrument;
3. Calibration.

For checking repeatability and daily and short-term stability, pilot milk samples with assigned values of adequate quality (stability, homogeneity, assigned values traceable to international standards or reference materials) are to be applied.

In addition, regular checks are recommended on:

1. Carry-over effect (all methods);
2. Linearity (all quantitative methods);
3. Zero-setting (all quantitative methods, if applicable);
4. Homogenization (infra-red methods making use of an homogenizer);

Detailed standard operation procedures for internal control shall be provided by the manufacturer/supplier of the milk analyser system.

For compositional parameters and somatic cell count, it is advised to fulfil requirements about frequencies and limits as listed in Table 5.

Table 5. Minimum frequencies and limits for internal controls.

Checks	Frequency	Fat / (True) Protein / Lactose (g/100 g)	Urea (mg/100 g)	Somatic Cell Count (%)*
Standard deviation of repeatability (sr)	Daily	0.014 (lab) 0.035 (on-farm)	1.4 (lab) 3.5 (on-farm)	4 (lab) 10 (on-farm)
Daily/short-term stability Standard deviation of intra-reproducibility (sRintra)	>3/hour (lab) daily (on-farm)	±0.05 (lab) ±0.12 (on-farm)	2.0 (lab) 5.0 (on-farm)	8.000 cells/ml (lab) 20.000 cells/ml (on-farm)
Zero setting	>4/day (lab) daily (on-farm)	±0.03 (lab) ±0.07 (on-farm)		8.000 cells/ml (lab) 20,000 cells/ml (on-farm)
Instrumental fittings (if applicable) Homogenization (percent)	Monthly	1 (lab)		Not applicable
Carry-over limit Lc (percent)	Monthly	1 (lab)		2 (lab)
Linearity ratio De/DL (percent)	Quarterly	1 (lab)	2 (lab)	2 (lab)
Calibration check Mean bias	Monthly	±0.05 (lab) ±0.12 (on-farm)		±5.000 cells/ml (lab) ±12.500 cells/ml (on-farm)
Calibration check Slope	Monthly	1.00 ± 0.03 (lab) 1.00 ± 0.07 (on-farm)		1.00 ± 0.05 (lab) 1.00 ± 0.12 (on-farm)

*If not differently specified

Note 1:

In case of high fat and protein concentrations (milk from buffaloes, sheep, and specific cow and goat species), adapted limits for repeatability and short-term stability can be set proportional to the average level for cow’s milk.

Note 2:

The indicated limits are based on the protocol for milk analyser verification for official milk recording purposes and ISO 8196-3|IDF 128-3.

Explanation with internal control checks

Repeatability: A repeatability check is indicating whether or not the analyser is working stable. Repeatability is evaluated at the start-up of each analyser system on the basis of 10 times replicate analysis of one (pilot) milk sample. During routine testing a regular test can be made by replicate analysis of a pilot sample. The estimate for the standard deviation of repeatability should meet stated limits.

Daily and short-term stability: Every day and regularly along a working day, pilot samples can be used to check instruments functioning at one or more concentration levels. Differences observed against assigned values should not exceed the stated limits.

Zero-setting: With infra-red instruments and some other analyser systems, rinsing the flow system and checking the "zero value" are periodically required to check for fouling on the walls of the measurement cells and/or (depending on instruments) to detect any drift of the basic signal.

Homogenization (only applicable with IR instruments): With analysis using infra-red instruments, the natural size of fat globules strongly affects the measurement of the fat content, therefore a fat size reduction is applied through an homogenization before the measurement. Inefficient homogenization results in poor repeatability and drifts of the signal.

Carry-over: In case of successive samples with strong differences of component concentrations, the result for a sample may have been affected by the former milk sample, e.g. by the residual volume of milk in the flow system or by the contamination by the stirrer and the pipette. The error is a proportion of the difference of concentration with the previous sample. The overall carry-over effect should be minimised, should not exceed limits stated and, when stable, can in routine operation be corrected for by applying carry-over compensation factors.

Linearity: Specific sets of samples to cover the whole range of concentration are required  so  that the instrumental measurement signal is proportional to the concentration of the component measured. The percentage of the bending can be estimated by the ratio (range of the residuals observed) x 100 /(range of the levels).

Mean bias: Representative milk samples are used to check the validity of the calibration at a medium level and to indicate whether any drift has occurred due to changes in milk composition or progressive wear of instruments.

Slope: Specific sets of (calibration) samples are prepared in order to cover the whole range of levels and check that the slope is within the stated limits. The larger the range of concentrations in routine samples, the bigger the error for extreme values in case of an inadequate slope adjustment.

External control

A periodical check on the accuracy must be applied through external comparison, either through individual comparison to reference analysis or through participation in proficiency testing. The minimum frequency recommended is 2 times a year with at least 12 samples per round.

Proficiency trials are to be organised in quality assurance conditions, according to international standards, or failing that, international guidelines or agreements:

1. ISO 17043  Conformity assessment — General requirements for the competence of proficiency testing providers
2. International Harmonized Protocol for Proficiency Testing of (Chemical) Analytical Laboratories. (IDF Bulletin 342:1999).
3. ISO 13528 Statistical methods for use in proficiency testing by interlaboratory comparison

Note:

In case of measurement of milk recording parameters with on-farm operated milk analyser systems, use can be made of herd bulk milk, sampled just before delivery to a dairy. Own testing results on the herd bulk milk can be compared with laboratory data from the milk payment sample. Such comparisons are to be conducted at least weekly, whereby the combined results over the last 12 milk collections continuously have to meet the limits as stated in Table 6. An example result sheet with the calculated values for the mean bias and the standard deviation of differences is contained in Table 7.

Table 6. Limits for external controls with use of milk analyser systems for milk recording parameters. Limits apply for every series of results from the last 12 subsequently milk collection herd bulk milk samples or pilot milk samples in singular, with analysis of at least one sample each three days.

Parameter	Fat / True Protein / Lactose (g/100 g)	Urea (mg/100 g)	Somatic Cell Count (%)**
Mean Bias	±0.05 (laboratory) ±0.12 (at-line/in-line)	±1.2 (laboratory) ±3.0 (at-line/in-line)	±5.000 cells/ml (laboratory) ±12.500 cells/ml (at-line/in-line)
Standard Deviation of Differences*	0.05 (laboratory) 0.11 (at-line/in-line)	5.0 (laboratory) 12 (at-line/in-line)	8 (laboratory) 20 (at-line/in-line)

*As reported below table 4 Standard deviation of differences is not the same asf Accuracy standard deviation s_y,x  

***If not differently specified        

Table 7. Example calculation for fat content for on farm milk analyser

Sample number	On-Farm Result (g/100g)	Lab Result (g/100g)	Difference (g/100g)	Mean Bias (g/100g)	Std Differences (g/100g)
1	4,17	4,11	0,06	–	–
2	4,15	4,07	0,08	–	–
3	4,05	4,09	-0,04	–	–
4	4,21	4,11	0,10	–	–
5	3,92	4,08	-0,16	–	–
6	4,05	4,05	0,00	–	–
7	4,00	4,04	-0,04	–	–
8	4,31	4,11	0,20	–	–
9	3,92	4,08	-0,16	–	–
10	3,92	4,10	-0,18	–	–
11	3,95	4,06	-0,11	–	–
12	3,95	4,10	-0,15	-0,33	0,12*
13	3,95	4,11	-0,16	-0,05	0,12*
14	3,92	4,08	-0,16	-0,07	0,12*
15	3,95	4,15	-0,20	-0,09	0,12*
16	4,25	4,06	0,19	-0,08	0,13*
17	3,95	4,20	-0,25	-0,09	0,14*
18	3,95	4,23	-0,28	-0,11	0,15*
19	3,97	4,23	-0,26	-0,13*	0,15*
20	3,98	4,23	-0,25	-0,16*	0,12*

*Non compliant with limit.

Keeping record of control data

Any user of a milk analyser system, whether being a laboratory or a user on-farm, should keep closely record of both internal control and external control checks. Date, milk analyser system identification, type of check, samples used for the check, test results and possible follow-up action should be recorded in files that are to be archived for a period of at least two years. These data are to be shared with external supervisors or auditors from the milk recording organisation or from any other thereto assigned entity upon their request.

Annex B Reference methods

Fat:	ISO 23318|IDF 249 - Milk, dried milk products and cream - Determination of fat content - Gravimetric method; or AOAC 989.05 - Fat in Milk: Modified Mojonnier Ether Extraction Method
Protein	ISO 8968-1|IDF 20-1 - Milk and milk products - Determination of nitrogen content - Part 1: Kjeldahl principle and crude protein calculation; or AOAC 991.20 - Nitrogen (Total) in Milk: Kjeldahl Methods
True protein	ISO 8968-1|IDF 20-1 - Milk and milk products - Determination of nitrogen content - Part 1: Kjeldahl principle and crude protein calculation; or AOAC 991.20 - Nitrogen (Total) in Milk: Kjeldahl Methods; and ISO 8968-4|IDF 20-4- Milk and milk products - Determination of nitrogen content - Part 4: Determination of protein and non-protein nitrogen content and true protein content calculation (Reference method); or AOAC 991.21 - Nonprotein Nitrogen in Whole Milk: Kjeldahl Method
Lactose	ISO 22662|IDF 198 - Milk and milk products - Determination of lactose content by high-performance liquid chromatography (Reference method)
Urea	ISO 14637|IDF195 - Milk - Determination of urea content - Enzymatic method using difference in pH (Reference method)
Somatic cell count	ISO 13366-1|IDF 148-1 – Milk - Enumeration of somatic cells - Part 1: Microscopic method (Reference method)