Skip to content

NovaStar Data / Data Types

Introduction

The NovaStar design includes the concept of "data type". However, due to traditional conventions with ALERT and other data, the NovaStar design has not handled data type as explicitly in the past as needed for web services. Whereas the NovaStar convention with ALERT/ALERT2 systems was to use a point numerical identifier corresponding to a sensor (and rely on the point's sensor sensor number, name and other descriptive information to understand the data type), web services use a string to indicate data type. The legacy NovaStar design has been adapted to support data types, as discussed below.

Database Data Type

The NovaStar 5 database design indicates data type in a number of ways, as summarized in the following table. Of the following options, the shef_parameter_code in point types and rating assigns is the most reliable to use for data type, as highlighted in bold in the following table.. See the next section for discussion of SHEF parameter code and limitations with using SHEF. The term "SHEF parameter code" refers to traditional National Weather Service SHEF parameter codes, "custom SHEF code" refers to custom codes within the SHEF specification, and "non-SHEF data type" or "custom data type" refers to data types that do not have a SHEF parameter code equivalent.

NovaStar Data that are Candidates for Providing Data Type

Database Column                                 Description Limitation
Point Type
point_type.description Often used to indicate data type. Verbose narrative text not suitable for data type.
point_type.name Often used to indicate data type. Verbose narrative text not suitable for data type.
point_type.numid Often used to indicate data type for legacy ALERT and may be used in ALERT2. For example a station may have numeric ID 100 and the precipitation point would have numeric ID 101, where 1 added to the station identifier indicates precipitation. Relying on convention only applies to some stations and protocols and is unreliable as a convention.
point_type.
shef_parameter_code		 Often used to indicate data type, required for SHEF import/export. Traditionally used for SHEF codes and non-SHEF data types. This is now also used for non-SHEF data types for web services.
point_type.short_name Often used to indicate data type. Could be used as data type but was implement for specific purpose and is in use on legacy systems.
Point
point.description Often used to indicate data type. Verbose narrative text not suitable for data type.
point.name Often used to indicate data type. Verbose narrative text not suitable for data type.
point.sensor_id Used for ALERT2 to indicate sensor for communication. Could be used to associate sensor type with data type but not generally applicable.
point.tag_name Often used to indicate data type, must be unique within a NovaStar system so may also include a unique point identifier. In the past was often set to the same as pointNumId. This is now being used to uniquely identify time series when multiple same data types are found at a station.
Rating Assign
rating_assign.
computation_type		 Indicates the computation type for ratings as a numerical code. Some computations are general, such as rating table, but others are specific (e.g., seasonal rainfall). These can be used to assume some specific data types such as seasonal rainfall but are ambiguous in most cases because the computation type is a general technique and does not indicate data type.
rating_assign.
shef_parameter_code		 Indicates the SHEF parameter code for values output from the rating. For example, input of accumulated precipitation (SHEF physical element PC) can be output as actual increment or bucket tip (PP), both as instantaneous values. Traditionally used for SHEF codes and non-SHEF data types. This is now also used for non-SHEF data types for web services.

SHEF Parameter Code

NovaStar uses a SHEF parameter code (shef_parameter_code database column) to indicate data type and duration for point scaled and rated values (discussed in more detail in the Data Reports and Time Series documentation). See also the Standard Hydrometeorological Exchange Format (SHEF) Manual.

The SHEF parameter code was not implemented in the original NovaStar design but was phased in later to facilitate exchange of data between neighboring systems and the National Weather Service (NWS). The use of SHEF for data exchange became common when ALERT2 became available because neighboring systems were configured to collect data in teach system and share some stations via SHEF. Prior to ALERT2, neighboring systems would often listen for stations in neighboring systems. With ALERT2, the convention is typically to let each system collect data for its stations and then export/import to share data, thus system configuration and maintenance issues for stations that are not actually owned.

There is no universal data type convention in various data collection systems. For example, the National Weather Service uses SHEF, the United States Geological Survey uses "parameter codes" and corresponding names, and other entities use other data type vocabularies, typically called "data types", "parameters", "variables", or similar. Consequently, because SHEF parameter codes are already used in NovaStar, the data have been adapted to provide flexibility for data types outside of the NWS definitions.

The SHEF parameter code is at minimum 2 characters and at most 7 characters and has the following parts, all of which are upper case:

PEDTSEP

SHEF Parameter Code Parts

SHEF Part Description Default
PE
required		 Physical element, generally equivalent to a data type. See Table 1 in the SHEF manual. For example QR is "discharge, river". Should always be specified.
D
optional		 Duration. See Table 3 in the SHEF manual. For example I indicates instantaneous, E indicates 5Minute, and H indicates Hour. I
TS
optional		 Type and source. See Table 4 in the SHEF manual. This is typically used for model data, flash flood guidance, or some other process. Z can be used for missing.
E
optional		 Extremum code. See Table 5 in the SHEF manual. The codes focus on maximum and minimum in a duration and cannot represent a full range of statistics. Use Z as missing.
P
optional		 Probability code. See Table 6 in the SHEF manual. The codes are used to indicate chance value is at or below a specified value. Use Z as missing.

The SHEF parameter code is designed to support NWS operations, but it is not a general-purpose "data type" code due to the following limitations.

  • The two-character physical element (PE) codes can be difficult to remember, and often do not intuitively match a data type due to the limited number of combinations (e.g., UD is wind direction and XR is relative humidity).
  • Predefined PE codes are not available for all data types that may be managed by a NovaStar system and there is limited flexibility for defining custom SHEF codes, which may lead to codes being defined differently on different systems.
  • Although it is possible to define custom SHEF codes, for example starting with Y, a custom SHEF code on one system (e.g., YB) may have a different meaning on another system and therefore the meaning of YB is ambiguous.
  • Handling of duration for time series has ambiguities (discussed in more detail below).
  • Extremum values are limited to minimum and maximum for some measurements and therefore limit use for other statistics. Data web services instead uses a "statistic" modifier for basic data types, coupled with the regular interval, for example WaterLevel-Max.1Hour

The SHEF standard is primarily used for exchange of real-time ("instantaneous") data, hence the default duration is I for instantaneous data. Consequently, the strict interpretation of the duration is the time prior to the measurement time. In a real-time event-based system such as NovaStar, this works well for the data reports and values calculated at the same timestamp (e.g., precipitation increment computed from accumulated precipitation and river discharge computed from water level),

However, the duration is ambiguous when applied to interval time series. For example, a SHEF parameter code of PPH indicates actual precipitation increment for the 1Hour duration prior to the timestep. There is no guarantee that the timestep occurs on exactly one-hour intervals, unless the system forces this constraint during calculations. For NovaStar, a rated accumulation can be configured; however, data imports for interval data (or data collection from a station that computes interval data in the data logger) could also be configured. The SHEF PEH does not require the times to be at regular interval.

In summary, SHEF parameter codes are suitable for use as data type in NovaStar primarily for points that are involved in SHEF format for import and export between systems, which is typically hydrometeorological data such as precipitation, wind, water level, etc. However, a more robust handling of data type is needed, and is discussed in the next section.

Data Web Services Data Types

As indicated in the previous sections, NovaStar as traditionally implemented does not fully handle defining data types. Legacy users typically have become familiar with referencing points equivalent, for example "point 101" means "station 100 precipitation". However, this is a limitation for web services, and applications that rely on web services, which need more easily understood data types.

The solution that has been implemented in NovaStar is to use the shef_parameter_code column in the point type and rating assign tables to support standard SHEF parameter codes and non-SHEF data types, as described below.

Guidelines for shef_parameter_code Values

Need to Share Data via SHEF? SHEF Parameter Code Available? Data Interval Recommendation for shef_parameter_code
Use SHEF parameter code
Yes Yes Instantaneous data. Use standard SHEF parameter code with or without duration I.
Yes Yes Instantaneous data with measurement over a duration, such as 1Hour precipitation total ending at the report time. Use standard SHEF parameter code with appropriate duration code.
Yes No Instantaneous or interval data. Could define a custom SHEF code such as YB, but using a non-SHEF data type (below) is more robust and avoids ambiguity with whether the interval aligns with regular intervals (or is a duration ending on the measurement time).
No Yes Instantaneous data. Use standard SHEF parameter code (since built in to web services) or see next option to define a non-SHEF data type.
Use non-SHEF data type
No No Instantaneous data. Use web services non-SHEF data type (DataType) and default irregular interval for instantaneous data is assumed (IrregSecond).
No No Interval data. Can use a standard SHEF parameter code but be aware of limitations discussed in previous section. Or, use a web services non-SHEF data type and interval such as DataType.1Hour.

The NovaStar system currently only uses the shef_parameter_code when importing and exporting SHEF data. Therefore, if data for a point is not going to be exchanged via SHEF, using a data web services non-SHEF data type is appropriate.

Regardless of approach for data type, data web services uses human-friendly data types for time series and data identifiers rather than SHEF physical element code. For example a SHEF physical element code of QR has a web services data type of DischargeRiver. For compatibility, where possible, built-in web services data types have been assigned that closely match SHEF physical element from the SHEF manual and these data types are the default built-in data types for a system. Additional web services data types are added over time as new requirements arise.

Important: Data web services examines the first two letters of the shef_parameter_code. If both are uppercase, then a SHEF parameter code is assumed. Otherwise, a non-SHEF data type is assumed.

See the following services to list available SHEF physical elements and data type:

  • dataTypes - data web services data types (includes matching SHEF physical elements)
  • shefPhysicalElements - SHEF physical elements without non-SHEF data types

Web services data types, statistics, and intervals are used in combination to uniquely identify time series, as shown in the following examples. These combinations are used in various web services.

Example Web Service Data Type Syntax

Data Type and Interval                                             Description
WaterLevelRiver Instantaneous river water level, equivalent to SHEF HG or HGI.
WaterLevelRiver.IrregSecond Instantaneous river water level, equivalent to SHEF HG or HGI. Support for interval in non-SHEF data types is under development.
WaterLevelRiver-Max.1Hour 1Hour interval time series maximum river water level, computed from the sample of data reports within the hour. Support for interval in non-SHEF data types is under development.