ARM projects at Mission Research

Objectives

1. Develop new data analysis and visualization tools that will improve ARM data utility.
2. Research and develop new automated software to detect data quality deficiencies in ARM data.

Tasks

1. Coordinate with the ARM Data Quality Office to identify data products requiring further quality control.
2. Develop better bounds for data values by: analyzing historical data, applying models, and consulting with experts.
3. Improve upon current min/max/delta quality criteria by implementing new rule-based, statistical based and cross-comparison based measures.
4. Develop new exploratory data analysis tools, specific to ARM data sets.
5. Develop a system that reassesses the quality of instrument level data streams based on information provided by value added products.

Approach

To help identify data quality issues with ARM data, we implement software to read existing quality control information from corresponding data streams, and consolidate this information into a form that can be displayed by the data quality office 'DQ HandS' system.

When we analyze a data stream without existing quality control (QC) information, we look at a representative sample of output files from the data archive to determine 'typical' values of the output variables. We then design outlier tests, specific to each variable, to be applied to future data. If a measurement is determined to be inconsistent with previous measurements (i.e., is an outlier), it is flagged as a possible data quality problem.

We supplement our QC analysis with various visualization tools that we make available to the ARM community.

These include IDL procedures that work with ARM data sets, and that generate plots that emphasize interesting features or data quality concerns.

We also develop Web based visualization tools that can be used from any Internet accessible computer, and that therefore make access to ARM data simpler, and more meaningful to a wider audience.