HOW WaDE 1.0 WORKED

The WaDE program espouses a principles-based approach to sharing data, with data curators maintaining their own datasets as the goal.

The WaDE program has taken a federated and distributed approach to data distribution. With this in mind, data are not transferred to a central location or repository, but instead remain housed within their agency or on a remotely hosted/cloud application controlled by the agency. Data retrieval is enabled using web services, which are interoperable, machine-readable, and capable of supporting a variety of applications. To enable data discovery, the WSWC maintains a “central catalog” of the data that is continuously updated.  The central catalog points the user to the state agency’s most recent data release.

  • WaDE seeks to make transparency, openness, and discoverability the default for public water data, while also ensuring security and privacy for stakeholders where that is required.

  • Data should be shared using developed standards and machine-readable format, including thoroughly documented metadata.

  • ata are shared using a federated approach where data-producers maintain control over and responsibility for updating and maintaining their information.

    The IT components for WaDE include databases, code to enable web services, and a map interface for data discovery. To learn more about how to access the web services without the central catalog or to develop an application, please see the Developers Section below.

     

  • WaDE APPLICATION

    GENERAL DOCUMENTATION AND GUIDELINE INFORMATION

    Current Central Catalog – July 16th, 2018 – Updated approximately every two weeks.

    Data Exchange Issues & Recommendations: The WaDE Schema/Technical Approach Workgroup has made recommendations for an approach to the data exchange that meets the needs of the states and supports the needs of the WaDE project.

    Western State Water Program Capabilities Assessment Survey Report – Final Draft: This document was developed by the State Capabilities Assessment Workgroup and is a compilation of survey results submitted by WSWC member states concerning their water planning and data hosting practices.

    Most WaDE components are open source and can be used to either install a WaDE application, or to understand how to access the data. To access the available components, see the WSWC’s GitHub Repository.

    WaDE Data Elements (v0.25)

    • WaDE Data Dictionary: A list of data elements and definitions exchanged via WaDE web services. (v0.25)
    • Data Exchange Template (DET) for the WaDE Schema: A descriptive document with detailed information for each element, expected formats, XML names, and its corresponding table/column in the WaDE database.
    • Data Exchange Template (DET) for GetCatalog: A descriptive document for the GetCatalog web service with detailed information for each element, expected formats, XML names, and its corresponding table/column in the WaDE database.
    • Data Exchange Template (DET) for GetMethod: A descriptive document for the GetMethod web services with detailed information for each element, expected formats, XML names, and its corresponding table/column in the WaDE database.
    • Current WaDE XML Schema Definition (XSD): An .xsd (schema) file that defines the expected format for the XML data exchange. This .xsd is the XML representation of the WaDE DET template. (Note: This file requires an XML viewer. XMLpad, a free .xsd viewer, is available for download here.

    WaDE Database Schema (v0.25)

    WaDE Web Services (v0.25)

    • WADE FCD v0.2 Final Draft: This document describes in greater detail and depth the exchange methods and components that a partner would need to implement to maintain compatible services and data as part of the WaDE data exchange, as well as how the users access and consume the data services.
    • For more information on the WaDE REST API, see below.

    WaDE REST API

    This web service queries the WaDE REST web services for the data that is shared by the organization or organizations within a specified region. The return includes reports (years of data), data category, datatype, and hyperlinks to web services to actually retrieve the specified data.

    • URL: /WADE/version_number/web_service_name/web_service_call_and parameters
    • Method: GET
    • URL Parameters
      • Required: loctype=[HUC/COUNTY/REPORTUNIT], loctxt=[id], orgid=[id, ALL], state=[id, ALL]
      • Optional: None.
    • Success Response: Data are returned in either XML format, or XML with a stylesheet applied to the data.
    • Error Response:
      • Code malfunction: Invalid response
      • Incorrect parameters: {error : “Connection to database successful, but parameters are invalid. Please check parameters.” OR
        • 404 Error – Page not found
    • Sample call: http://www.westernstateswater.org/CentralCatalog/WADE/v0.2/GetCatalog/GetCatalog.php?loctype=REPORTUNIT&loctxt=2&orgid=ALL&state=47
    • Notes: The GetCatalog web services retrieves information about the data shared for a particular indexed region at a time. To retrieve more than one region at a time, the GetCatalog_GetAll web service can be used. To request this, please join the WaDE community forum and send a request to the administrator.

    This web service queries the WaDE REST web services for the summarized, aggregated data that is shared by the organization or organizations within a specified region. The return includes a summary for the year requested, either by datatype, or for all datatypes, and hyperlinks to web services to retrieve methodology information.

    • URL: /WADE/version_number/web_service_name/web_service_call_and parameters
    • Method: GET
    • URL Parameters
      • Required: loctype=[HUC/COUNTY/REPORTUNIT], loctxt=[id], orgid=[id], reportid=[id], datatype=[id, ALL]
      • Optional: None.
    • Success Response: Data are returned in either XML format, or XML with a stylesheet applied to the data.
    • Error Response:
      • Code malfunction: Invalid response
      • Incorrect parameters: {error : “Connection to database successful, but parameters are invalid. Please check parameters.” OR
        • 404 Error – Page not found
    • Sample call: http://www.westernstateswater.org/Wyoming/WADE/v0.2/GetSummary/GetSummary.php?loctype=REPORTUNIT&loctxt=2&orgid=WYWDC&reportid=2014&datatype=ALL
    • Notes: None.

    This web service queries the WaDE REST web services for the detailed, site-specific data that is shared by the organization or organizations within a specified region. The return includes tables for the year requested, by datatype, and hyperlinks to web services to retrieve methodology information.

    • URL: /WADE/version_number/web_service_name/web_service_call_and parameters
    • Method: GET
    • URL Parameters
      • Required: loctype=[HUC/COUNTY/REPORTUNIT], loctxt=[id], orgid=[id], reportid=[id], datatype=[id, ALL]
      • Optional: None.
    • Success Response: Data are returned in either XML format, or XML with a stylesheet applied to the data.
    • Error Response:
      • Code malfunction: Invalid response
      • Incorrect parameters: {error : “Connection to database successful, but parameters are invalid. Please check parameters.” OR
        • 404 Error – Page not found
    • Sample call: http://www.westernstateswater.org/WYSEO/WADE/v0.2/GetDetail/GetDetail.php?reportid=2015&loctype=REPORTUNIT&loctxt=2&datatype=ALLOCATION
    • Notes: None.

    This web service queries the WaDE REST web services for methodology information that is shared by the organization or organizations. The return includes tables for either all methods used by the organization, or one methods requested, and hyperlinks to data sources and any additional information hosted by the organization concerning methods used to generate the data.

    • URL: /WADE/version_number/web_service_name/web_service_call_and parameters
    • Method: GET
    • URL Parameters
      • Required: methodid=[id, organization_identifier], methodname=[id, ALL]
      • Optional: None.
    • Success Response: Data are returned in either XML format, or XML with a stylesheet applied to the data.
    • Error Response:
      • Code malfunction: Invalid response
      • Incorrect parameters: {error : “Connection to database successful, but parameters are invalid. Please check parameters.” OR
        • 404 Error – Page not found
    • Sample call: http://www.westernstateswater.org/Idaho/WADE/v0.2/GetMethod/GetMethod.php?methodid=IDWR&methodname=ALL
    • Notes: None.

     

FOR DEVELOPERS

  • To build an app using R Shiny, you will first need to load the shiny library, and then create one script named ui.R, and another named server.R. These scripts must be located in the same directory in order to run the application. The ui.R script will contain information about the layout, display, and individual output features. The server function contains the commands for generating the output (e.g. creating a plot, calculations, etc). The complete process for setting up the scripts is detailed in this article and in a series of video tutorials.
  • Both the ui.R and the server.R files for this application are available on GitHub. For this example, the ui.R script contains a titlePanel which displays the text “California Water Supply and Water Use Data Exploration”. Other features are contained in the sidebarPanel (including text features explaining the purpose of the app, and slider and dropdown selection tools, (sliderInput and selectInput)) and the mainPanel contains placeholders (similar to div tags in HTML) for the plots and outputs (plotOutput and textOutput) which will be created in the server.R script. The ui.R script also references the possible reporting units available for the user to choose via the selectInput feature.The server.R script loads the necessary libraries for reading in the data from the WaDE web service (library(XML)) and formatting plots (library(ggplot2) and library(RColorBrewer). The script contains a function for fetching the reporting units using the GetCatalog_GetAll web service, as well as a function with commands that are reactive to the user input (such as the year which is selected in the sliderInput feature), perform parsing and formatting tasks, and create the outputs which are displayed in the containers specified in the ui.R script (such as a bar plots which are displayed in the plotOutput containers).
  • The WaDE GetSummary web service is called using the xmlParse command from the XML library. The actual URL will follow the structure of the web service, treating the reporting unit and year as reactive variables determined by the inputs (input$reportingunit and input$year) which are defined by the slider and dropdown selection features. xmlRoot and xmlSApply are then used to parse through the XML file and search for the desired values, such as the types of water use, and the amount. The renderplot function contains the commands for creating and formatting the plot of the values from the XML data and assigns it to a specific container.The WaDE GetMethod web service is also called and parsed to obtain desired values, such as the URL for the corresponding method. This value is displayed as text using the renderText function.