Table of Contents
...
Outline of the overall multi-agency architecture
Description of the data model and API standard through which all agency data will ideally be integrated and served to users
Description of options for how agencies can provide their data in the standard data model and API standard.
...
Currently, many (but not all) agency data are already published online through services such as ESRI web maps, excel files, or in some cases public APIs. However, important aspects for a given data type (such as water table level measurements from wells) such as data/time formats, geospatial projections, column names, and units vary from agency to agency and even from dataset to dataset within agencies. In order to allow users to access data from multiuple agencies in one format, the NMWDI architecture will route all agency data through one Web API standard with one corresponding underlying data model that references one common statewide water data controlled vocabulary. As long as each agency somehow serves their data through the common Web API, data storage can be federated (i.e. not centralized), although some degree of centralization can be accomodated if that is most convenient for a given dataset. Each agency’s standardized API will be published through a central portal with an NMWDI administered API Management Platform. Users can send API requests to the management platform, which will route these requests to the agency APIs and in turn forward the responses to users. However, whether data storage is federated across agencies or centralized, all contributing agency data will be required to be mapped to the common data model amd controlled vocabulary and transformed into the common format before being delivered to users. This basic data flow is illustrated in Figure 1.
...
SensorThings Entity | Description | Example: NMBGMR Aquifer Monitoring Well | |||
|---|---|---|---|---|---|
Metadata | Location | A unique coordinate or area on the surface of the earth | Location in latitude and longitude or UTM easting and Northing (UTM Zone 13, NAD83) | Street Address (possibly with associated latitude and longitude). (e.g. 3960 PRINCE ST) | Location in easting and northing (UTM NAD83 in meters) |
Thing | Some real-world thing with which one or more Sensors are associated | Well Point ID WL-0150 | Sample Pt RT236I | Point of Diversion POD Number A 00008 AS | |
Datastream | A collection of Observations about an ObservedProperty produced by a Sensor associated with a Thing | Time series, Hydrograph | Sample Results | Meter Readings (Quarterly) | |
Datastream/observationType | The type of observation, codified in the Observations and Measurements data standard. Types include Categorical (defined text), Count (integer), Measurement (continuous number), Observation (free text), and TruthObservation (True/False) | Measurement | Categorical or TruthObservation | Measurement | |
Datastream/unitOfMeasurement | A three-item definition of the unit of measurement, including its name, symbol, and link to the definition (preferably to one provided in an established ontology such as http://unitsofmeasure.org/ucum.html or http://qudt.org/) | feet (e.g. http://qudt.org/vocab/unit/FT) | TCR Result | Acre-Feet (e.g. http://qudt.org/vocab/unit/AC-FT) | |
Sensor | The procedure used to provide a Datastream. Can be a particular data recording device model, or a defined procedure followed by a human observer. If applicable, a specific instance (e.g. a sensor model and serial number) | Steel-tape measurement; Continuous acoustic sounder | 9223B-PA (https://www.standardmethods.org/doi/10.2105/SMWW.2882.194) | MCCROMETER Diversion Meter-Meter Number 17147 | |
ObservedProperty | The raw or processed phenomenon (quantitative or qualitative) being measured for the Datastream. Preferably including a link to a definition provided by an established ontology or controlled vocabulary such as the ODM2 Controlled Vocabularies or http://qudt.org/) | Depth to Water Below Ground Surface (BGS) | Analyte (e.g. Coliform (TCR) (3100)) | Mtr Amount | |
OPTIONAL: FeatureOfInterest | The real-world feature that the Observations are about. This may or many not be different from the Location where the Thing on which the Sensor is mounted. Can include a JSON-formatted point location or a polygon or collections thereof. | Formation (e.g. https://maps.nmt.edu/maps/data/hydrograph/formation_lu) | Public Water System (head office location or service area boundary) (e.g. Albuquerque Water System PWSID NM3510701) | Water Right (set of relevant points of diversion) | |
Data | Observation | A single measurement value including the result, time values, and other metadata. Information on the ObservedProperty that was measured by what Sensor is provided by the Datastream these observations are in. Features of Interest are linked for each observation as well. Observations are linked to (collected in) Datastreams | Depth Measurement | Sample (e.g. 763391) | Meter Reading |
Observation/result | The actual measured value, with valid values defined in observationType and units defined in unitsOfMeasurement, both provided by Datastream | Depth (e.g. 337.08) | Sample Result (P (Positive/ Coliform found) A (Negative/ Coliform not found)) | Mtr Amount (e.g. 107.948) | |
Observation/phenomenonTime | The date+time (or interval) in ISO 8601 format (YYYY-MM-DDT:HH:MM:SS-Z) when the observation occured | 2019-01-31 00:00:00 | MP (Monitoring Period) (e.g. 01-01-2020 to 01-31-2020) | 1/20/2017-04/05/2017 (Quarterly period for which volume was measured) | |
OPTIONAL: Observation/resultTime | The date+time that the result was generated. May be the same as phenomenonTime | Date (e.g 01-06-2020) | 04/05/2017 (date of meter reading) | ||
OPTIONAL: Observation/validTime | The date+time interval during which the Observation can be used (often used for provisional values that are replaced by QA/QC’d observations) | ||||
OPTIONAL: Observation/resultQuality | A description of the result Quality. Will vary according to agency practice. Can use ODM2 controlled vocabulary for data quality types as a guide. | Precision (e.g. “within two hundredths of a foot”) |
...
Set up a SensorThings database and API instance in a preferred environment, configuring its domain name and security as appropriate to the agency.
Write scripts to Extract, Transform, and Load (ETL) data from existing databases or tabular data sources (e.g., csv, excel) and upload them to the SensorThings database using the SensorThings API.
Provide the URL of the agency SensorThings instance to NMWDI for registation in the API Manasgement Management platform and CKAN.
Step 1: Setting up a SensorThings instance
...
SensorUp is the reference implementation of STA. It is a private Canadian company offering a license + subscription model with customer support and an integrated dashboard and visualization suite. SensorUp is lead by Dr. Steve Liang, who was the lead developer of SensorThings API.
FROST-Server, a free and open-source server written in Java designed for deployment in a Tomcat Java Servlet with a PostGIS database. FROST is developed by the Fraunhofer Society, the main government-supported applied research organization of Germany.
52 North STA, a different free and open-source server written in Java designed as a Spring Boot service with a PostGIS database. 52 North is a non-profit geospatial IT research entity associated with German and EU universities and research consortia that develops open source software as well as provides professional IT consulting.
Geodan GOST, a free and open-source server written in Go designed as a platform-independent Go service connected to a PostGIS database and a basic graphical user interface. Geodan is a private Dutch firm specializing in geospatial business analytics.
All three four options can be made to run through cloud-native stacks such as Google App Engine or through any local physical or virtual machine environment through containerization paradigms such as Docker and Kubernetes. The fastest way to set up a simple SensorThings instance is by using Docker and Docker Compose. Docker containers are virtual environments that can run on any host operating system and contain only the software and resources necessary to run a desired application. Docker Compose is a framework to run multiple Docker containers that are linked together in a virtual network on the same host. Below is a a simple process for setting up SensorThings API on a local machine or a cloud VM using FROST-Server Docker Containers.
...