for this first iteration assume uploading a single dataset, i.e. a time series of water levels from one well or stream gauge. A bulk multiple point upload will be developed later.
Starting point. I am a data provider and have made the required upload.yml file. Upload Format
Digression: WDI potentially should provide tools or guidance or the option to upload a non-normalized xls or csv file. This may be more conducive to agencies existing work flows. However the toolchain we are designed here is assuming the starting point is upload.yml
2. Upload. Login into the WDI upload app. submit your upload.yml file
another option is provide a web form at this point that will generate a meta.yml file for the user.
the user could select from user defined meta.yml.templates. These templates would simply prepopulate the entry form if selected, so the user could easily override existing settings.
3. upload.yaml loaded into Clowder Icebox.
clowder is just another Open Data portal. Its relatively lightweight but also seems to provide all the functionality we need for a staging area. CKAN will still serve as the public facing data catalog. Clowder will be used manage the internal ETL processes.
4. WDI QC? move data from Icebox to CKAN
the move would be handled by a dedicated “extractor”. triggered manually or maybe via tagging event?
5. WDI triggers a ST extraction.
extractor reads in upload.yaml and uploads records to a ST instance via the ST API.
setup a dockerized extractor image. launch a cluster of extractor containers.