Help Page
The LWI is using EnDMC as the data and model repository to promote the discoverability, reproducibility, and reuse of existing models. The term catalog is used across this site—and in place of repository—to denote the central purpose of the system: making a curated, discoverable list of resources.
Metadata is an integral piece of this catalog. Metadata is structured information, understandable by humans and machines, that describes, explains, and makes it easier to retrieve, use, and manage an information resource included in the data and model catalog. As such, metadata enables the resources included within the catalog to be discoverable, trusted, replicable, and reusable.
Metadata should be created by the person who is responsible for submitting a specific resource to the catalog. This individual is the creator, collector, or expert user of that resource (e.g., specific dataset or model) and is therefore familiar with the specific resource and can provide a description of it. This guide is intended for metadata authors, who have the responsibility to create the high-quality metadata needed to make the catalog useful.
To create metadata, please use the “Create” tab from the top menu. Note that you will need an editor role in order to create metadata. Below is a summary of the metadata specifications for models that were developed by The Water Institute.
To review existing metadata files, please use the “See all” tab from the top menu and select the type of metadata you are looking for.
To upload your model/simulation files, go to the “See all” tab and identify the metadata for which you want to upload files. Open it, scroll to the bottom of the page, and drag and drop your file in the “resource file” section.
To search within the repository, please use the “Search” tab from the top menu and use keywords to search for the best model that fits your need. You can use the filters on the left to customize your search.
Metadata Structure
Typical resources included in a data and model catalog include datasets, models, and model simulations. Given the complexity of some resources included in EnDMC (e.g., groups of numerical simulations developed with different software), it can be helpful to structure the metadata as a series of layers. A specific layer can be used for multiple resources, allowing the reuse of information, and reducing the time needed to generate complex metadata. Resources are related, and the presented metadata structure captures these relationships as shown in Figure 1.
Figure 1. Schematic representation of the layered approach for metadata, resource types and their relationships.
Datasets
For Datasets, the Data Catalog Vocabulary (DCAT) metadata standard was adopted. DCAT is a widely used World Wide Web Consortium (W3C) recommendation which allows for systems interoperability. This makes datasets indexable in specialized search engines and enables the use of catalog resources as part of federated systems. For datasets, only one layer of metadata was deemed necessary.
- Dataset: A logical unit with proper identity. A dataset can be composed of raw or preprocessed data in any format. It is intended to be reused for analysis or as inputs for a model.
Models
As there is no current standard with which to define metadata for numerical models, the design of the metadata for numerical models used in this catalog was informed by existing initiatives and interviews with modelers. Five different layers of metadata were identified to describe numerical models and numerical model simulations. This layered structure is designed to reduce duplicative work, enabling users to link to existing metadata pieces when appropriate. For example, if there are multiple simulations performed with the same model application, metadata for these simulations can all refer to the same model application metadata and only the specific differences of each simulation can be highlighted in the simulation metadata (e.g., the model grid will be defined at the model application level, the meteorological inputs at the simulation level). In addition to the metadata fields listed, a unique identifier is generated for each metadata record. Figure 2 shows the five layers of the Metadata Creation Tool that were adopted to describe models and their content.
Figure 2. Diagram shows five layers of the Metadata Creation Tool and gives examples of the level of information that needs to be captured at each layer.
- Software: Generic description, purpose, and characteristics of a software (e.g., ADCIRC, HEC-RAS, Delft3D). This
layer is version independent. It does not have an associated resource file or distribution package link, as these
elements will be included in the following layer (i.e., Software Version).
The catalog already includes a library of commonly used softwares. Only admin users can generate new software metadata. If the required software is not already included, please contact us at endmc@thewaterinstitute.org. The common fields between software and software versions are intended to be inherited. A software version will always have a singular software as its parent field. - Software Version: Describes the characteristics of a specific software version (e.g., HEC-RAS 4.9). It includes
information about its distribution, such as links to the container image, GitHub release, or distributor website and
installation instructions. It is linked to its software.
The repository already includes a library of software versions. Only admin users can generate new software version metadata. If the required software version is not already included, please contact us at endmc@thewaterinstitute.org. - Model Application: A model application that was built with a single software to simulate a specific environmental process or event (e.g., water transport, sediment transport) in a specific area (e.g., Barataria basin). It can be linked to a software version; if not, please include the “software version” field at the simulation layer.
- Simulation: A specific numerical simulation performed with a specific model application (e.g., validation run for a specific model application). A simulation is linked to a parent model application and to a specific software version (unless that link is provided at the model application layer). It requires a description of all input and output files (common input or output files across multiple simulations can be described at the common model application layer).
- Package: A group of simulations or a group of model applications that are grouped together because they are part of the same work (e.g., a set of simulations that are part of one production run/alternative) or of the same modelling effort (e.g., HEC-HMS and HEC-RAS for the same watershed). A package allows a user to define groups of simulations across multiple applications.
Detailed Field Information
The tables in this section provide a detailed description for the different layers of metadata used to describe numerical models, simulations, and datasets. To fully describe one model simulation, four layers are required: software, software version, model application, and simulation metadata. The simulation package layer is optional and required only if two or more simulations or model applications are part of a package that the metadata author wants to define.
The metadata author should use this guide to understand what information is entered into each layer and consider the complexity of the new resources that they are describing and try to leverage the layer approach to minimize the new metadata that needs to be created. The metadata author can use existing software metadata, software version metadata, and model application metadata if they are available and appropriate to a new resource (e.g., model application or simulation).
Some additional recommendations that the metadata author should keep in mind are:
- Each keyword can contain more than one word. Words or phrases that are commonly used when searching for resources should be used. If there is an acronym or a common name not mentioned in the title or the description, the keywords field is a good place to include this information.
- In the Organization Name fields, entire names are preferred to acronyms. However, acronyms can be used in the descriptions and keywords but should be defined in the description. This will allow a user to successfully search for datasets using both full names and acronyms.
- Clear titles and meaningful descriptions will positively impact the usability of the system.
- If a field is optional there will be a check box in front of it for selection.
Models
Software
The intent of this metadata layer is to provide a generic description and state the purpose and characteristics of a software (e.g., ADCIRC, HEC-RAS, Delft3D). This layer is version independent. It does not have an associated resource file or distribution package link, as these will be added in the software version layer.
To create a software metadata, select “Create/Software” from the navigation menu and then add the information detailed in Table 1.
| Field | Description |
|---|---|
| Load an existing metadata file | This link can be used to upload an existing software metadata, edit it, and if needed reuse it for new resources. This step can be ignored if creating a new metadata record. |
| Title | Human-readable name of the software. It should be in plain English and include sufficient detail to facilitate search and discovery, (include who, what, where, when, and scale). It is recommended providing consistent naming structures across the metadata records for a project or program. |
| Access Level | The degree to which this dataset could be made publicly available, regardless of whether it has been
made available. Three options are available:
|
| Model type | Select the type of model this type of software allows you to develop from the following list:
|
| Description | Brief description of the software. |
| Purpose | Brief explanation of the primary use case for this software (e.g., storm surge, water quality, etc.). |
| Authors (optional) | List of the primary software creator(s). To add an author, select the check box, which will prompt you for the contact’s name and email. Make sure to include the full email address of the author for future reference. |
| Contributors (optional) | List of additional contributors that participated in the software development. If there are any other contributors to the software that need to be included, select the check box, which will prompt you for the contact’s name and email. Make sure to include the full email address of the author for future reference. |
| Model category | Select the category or categories of model(s) that you can build with this software:
|
| Keywords | List of keywords or short phrases to help users discover your software. This field is required. Please include terms that would be used by technical and non-technical users. This field will prompt choices as the user types to limit the keyword variations in the final database. Multiple keywords can be entered by using the “+” button. |
Software Version
The intent of this metadata layer is to describe the characteristics of a specific software version. It should include information about its distribution, such as links to the container image, GitHub release, or distributor website and installation instructions. A software version will define the required, and optional, input and output files.
The software version used to run each simulation is specified in the simulation metadata. If all simulations of a specific model application use the same software version, this can be defined at the application level instead of at the simulation level.
The fields that are in italics below are inherited from the software metadata.
To create a software version metadata, select “Create/ Software Version” from the navigation menu and then add the information detailed in Table 2.
| Field | Description | ||
|---|---|---|---|
| Load an Existing Metadata File | This link can be used to upload an existing software version metadata, edit it and, if needed, re-use it for new resources.
This step can be ignored if creating a new metadata record. |
||
| Software | This is the general software this software version belongs to (i.e., HEC-HMS Version 2 belongs to HEC-HMS). Start typing the software title and a list of available options to choose from will appear. | ||
| Title | Name of the software version. This will auto-populate from the software. | ||
| Access Level | The degree to which this dataset could be made publicly available, regardless of whether it has been
made available. Three options are available:
|
||
| Model type | Select the type of model this type of software allows you to develop from the following list:
|
||
| Description | Brief description of the specific software version, what it does, and what its uses are. | ||
| Purpose | Brief explanation of the primary use case for this software version. | ||
| Authors (optional) | List of the primary software version creator(s).
To add an author, select the check box, which will prompt for the contact’s name and email of the author. Make sure to include the full email address of the author for future reference. |
||
| Contributors (optional) | List of additional contributors that participated in this software version development.
If there are any other contributors to the software that need to be included, select the check box. This will prompt you for the contact’s name and email of the contributor. Make sure to include the full email address of the contributor for future reference. |
||
| Model category | Select the category or categories of model(s) that you can build with this software:
|
||
| Keywords | List of keywords or short phrases to help users discover your software version; please include terms that would be used by technical and non-technical users. | ||
| Status Tag | Draft, Published, Required or Obsolete. | ||
| Components (optional) | Only for coupled models. List of the IDs of the model's components. | ||
| Parameters Description (optional) |
List of parameters that can be specified either through the command line or within a configuration file during execution. To add a parameter and description, select the check box and add a row. Enter the title, description, where to define it, and what the parameter’s default value is for the standard variable. |
||
| Input Files |
Description of the input files required by this software version. Optional input files can also be listed. To add an input file, select the check box and add a row. Enter the title, description, file prefix, file extension, what the parameters default value is for the standard variable and if it is required or optional. |
||
| Outputs Files |
Description of the output files generated by this software version. To add an output file, select the check box and add a row. Enter the title, description, file prefix, file extension, what the parameters default value is for the standard variable. |
||
| Distribution (optional) | Information on how and where to obtain the software distribution. To enter distribution information, select the check box then include the following information:
|
||
| Installation Instructions (optional) | Instructions for how to install this specific software version. Links to external resources can also be included. | ||
| Release notes (optional) | Any additional information, including version types. For example, details on new capabilities added in the new release or issues resolved from the previous software version. |
Model Application
The intent of this metadata layer is to describe a model application that was built with a specific software (described in the software version layer) and designed to simulate a specific environmental process (e.g., water transport, sediment transport) in a specific area (e.g., Barataria Basin).
To create a model application metadata, select “Create/Model Application” from the navigation menu and then add the information detailed in Table 3.
| Field | Description | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Load an Existing Metadata File | This link can be used to upload an existing model application metadata, edit it and, if needed, re-use it for new resources. This step can be ignored if creating a new metadata record. |
||||||||
| Title | Human-readable name of the model application. It should be in plain English and include sufficient detail to facilitate search and discovery, (include who, what, where, when, and scale). It is recommended providing consistent naming structures across the metadata records for a project or program. | ||||||||
| Description | Brief description of the model application, specifically what geographic area it was calibrated to and what processes or events are simulated. | ||||||||
| Purpose | Brief explanation of the primary use case for this model application (e.g., why was this model application developed? What is its goal?) | ||||||||
| Grid Description (optional) |
Description of the numerical grid used in this model application.
To add a grid description, select the check box then include the following information:
|
||||||||
| Spatial Extent (optional) |
The entire geographic area covered by the model application.
You can list the place name(s) and use the autocomplete, or add a HUC code, or add a custom WKT (you can define a bounding box using a custom WKT string with a service like this OpenStreetMap Tool). |
||||||||
| Intended Usage Area |
The geographic area where the model application was calibrated and validated. This is the area where the model
application is considered valid. If it coincides with the Spatial Extent, you can skip this field.
You can list the place name(s) and use the autocomplete, or add a HUC code, or add a custom WKT (you can define a bounding box using a custom WKT string with a service like this OpenStreetMap Tool). |
||||||||
| Temporal Extent (optional) | The period of time for which this model application is valid, start and end dates. | ||||||||
| Application Date | When the model application was created. | ||||||||
| Authors (optional) | List of the authors who developed or contributed to the development of this model application.
To add an author, select the check box, which will prompt for the contact’s name and email of the author. Make sure to include the full email address of the author for future reference. |
||||||||
| Common Software Version |
A link to the software version metadata can be added here if all simulations performed with this model application
used the same software version. The simulation metadata will inherit specific information from the common software
version.
Start typing the software version of interest and select from the list of available versions. |
||||||||
| Common Input Files (optional) |
List of all input files that all simulations performed with this model application have in common. The list will
automatically populate after selecting the software version (if a common software version was selected and if input
files were included at the software version layer). Add or remove input files as needed by clicking the + item button or
the delete button.
Enter the title, description, source of the dataset (e.g., if the input file includes data, specify where they came from), and location (e.g., file name, or file prefix, or file extension). |
||||||||
| Common Output Files (optional) |
List, if they exist, of all output files that all simulations performed with this model application have in common.
The list will automatically populate after selecting the software version (if a common software version was selected and
if output files were included at the software version layer). Add or remove output files as needed by clicking the +
item button or the delete button.
Enter the title, description, and location (e.g., file name, or file prefix, or file extension). |
||||||||
| Common Parameters (optional) |
List of parameters that all simulations performed with this model application have in common.
The list will automatically populate after selecting the software version (if a common software version was selected and if parameters were included at the software version layer). Add or remove parameters as needed by clicking the + item button or the delete button. For each parameter, include name and value. |
||||||||
| Linked Resources (optional) |
Additional helpful resources (e.g., reports, publications, websites) related to this model application. If a report
is available, please make sure to include it. Note: this is not required, but many future users use this field and find it helpful, so it is highly recommended. |
||||||||
| Model File External Location (optional) |
Link to an external location (e.g., website, S3 link) where the model application files are stored.
Please use this field only if for some reason you cannot upload the model files directly into EnDMC (see file upload section). |
Simulation
The intent of the Simulation metadata layer is to describe a specific numerical simulation performed with a specific model application (e.g., validation run for a specific application). A simulation is linked to a parent model application and to a model software version. It requires descriptions of input and output files.
To create a simulation metadata, select “Create/Simulation” from the main menu in the metadata creation tool and then add the information detailed in Table 4.
Note: In this version of EnDMC, you will need to define input and output files within the simulation metadata (or the model application in case they are shared across multiple simulations). It is not required to create dataset metadata for each input and output file unless the metadata author thinks it can add valuable information not contained in the simulation/application/software metadata.
| Field | Description |
|---|---|
| Load an Existing Metadata File |
This link can be used to upload an existing simulation metadata, edit it and, if needed, re-use it for new resources.
This step can be ignored if creating a new metadata record. |
| Title | Human-readable name of the simulation. It should be in plain English and include sufficient detail to facilitate search and discovery, (include who, what, where, when, and scale). It is recommended providing consistent naming structures across the metadata records for a project or program. |
| Simulation Description | Brief description of the simulation goals and main setting. |
| Software Version | The software version used to run this simulation. When typing, the list of available software version metadata will pop-up to allow the user to select the correct one. |
| Model Application | The model application used for this simulation. When typing, the list of available model application metadata will pop up to allow the user to select the correct one. |
| Simulation Type | The simulation category (e.g., validation, calibration, scenario, run). |
| Input Files |
List of all input files used for this simulation. The list of input files will automatically populate after selecting
the software version and/or model application (if common input files were included at one of those layers). Add or
remove input files as needed by clicking the + item button or the delete button.
Enter the title, description, source of the dataset (e.g., if the input file includes data, specify where they came from), and location (e.g., file name, or file prefix, or file extension). |
| Output Files |
List of all output files generated by this simulation. The list of output files will automatically populate after
selecting the software version and/or model application (if common output files were included at one of those layers).
Add or remove output files as needed by clicking the + item button or the delete button.
Enter the title, description, and location (e.g., file name, or file prefix, or file extension). |
| Parameters |
List the parameters used in this simulation.
The list of parameter files will automatically populate after selecting the software version and/or model application. To add or remove an input, click the + item button or the delete button. Enter the title, description, where to define it, and what the parameters default value is for the standard variable. |
| Temporal Extent (optional) | The period of time for which this simulation is valid. Use Date1 as the Beginning Date and Date 2 as the End Date. If there is no temporal extent on the model simulation, leave blank. |
| Temporal Resolution (optional) | Frequency at which the simulation results are saved. |
| Linked Resources (optional) | Additional helpful resources (e.g., report, publications, websites) related to this simulation. If a report is available, please make sure to include it. Note: this is not required, but many future users use this field and find it helpful, so it is highly recommended. |
| Model File External Location (optional) |
Link to an external location (e.g., website, S3 link) where the simulation files are stored.
Please use this field only if for some reason you cannot upload the model files directly into EnDMC (see file upload section). |
Package
The intent of this metadata layer is to describe a group of simulations or model applications that are part of the same work (e.g., a set of simulations that are part of one production run/alternative; HEC-HMS and HEC-RAS applications for the same watershed). A package allows, for example, a user to define groups of simulations across multiple applications.
To create a package metadata, select “Create/Package” from the main menu in the metadata creation tool and then add the information detailed in Table 5.
| Field | Description |
|---|---|
| Title | Human-readable name of the package. It should be in plain English and include sufficient detail to facilitate search and discovery, (include who, what, where, when, and scale). It is recommended providing consistent naming structures across the metadata records for a project or program. |
| Description | Description of the package, including an explanation of what the simulations or applications listed in this package have in common and what they represent. |
| Linked Resources (optional) |
Additional helpful resources (e.g., reports, publications, websites) related to this package. If a report is
available, please make sure to include it. Note: this is not required, but many future users use this field and find it helpful, so it is highly recommended. |
| Simulations or model applications | Connections to the simulations or the model applications that make up this package. A list to choose from will be prompted when typing. |
Datasets
A dataset is intended as a set of resources that can be described as a logical unit with proper identity. It can be raw or preprocessed data in any format that is intended to be reused.
The dataset metadata tool adopted the Data Catalog Vocabulary (DCAT) vocabulary. The definition of required fields is based on the DCAT-US specification, adapted to meet the needs of the catalog. Table 6 presents the fields included to describe each dataset.
To create dataset metadata, select “Create/Dataset” from the navigation menu and then add the information detailed in Table 6.
| Field Name | Description | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Load an Existing Metadata File |
This link can be used to upload an existing dataset metadata, edit it and, if needed, re-use it for new resources.
This step can be ignored if creating a new metadata record. |
||||||||||||||||||
| Title | Human-readable name of the resource. It should be in plain English and include sufficient detail to facilitate search and discovery, (include who, what, where, when, and scale). It is recommended providing consistent naming structures across the metadata records for a project or program. | ||||||||||||||||||
| Description | Human-readable description (e.g., an abstract) with sufficient detail to enable a user to quickly understand whether the asset is of interest. Please make sure to include the types of fields used, units, and any other details that may not be ascertained without studying the dataset at length. | ||||||||||||||||||
| Public Access Level |
The degree to which this dataset could be made publicly available, regardless of whether it has been made
available. Three options are available:
|
||||||||||||||||||
| Rights (optional) | Information regarding access or restrictions based on privacy, security, or other policies. This should also provide an explanation for the selected "Public Access Level" including instructions for how to access a restricted file, or explanation for why a "non-public" or "restricted public" data asset is not "public." | ||||||||||||||||||
| Frequency (optional) | Temporal frequency with which the dataset is published. | ||||||||||||||||||
| Project Point of Contact |
Contact person that can be contacted for additional information on this dataset. If the dataset entry is published
as “public” then it will be the contact person displayed on the public-facing entry.
|
||||||||||||||||||
| Data Dictionary (optional) | URL to the data dictionary for the dataset or API (e.g., National Hydrography Dataset Data Dictionary). Note that documentation other than a data dictionary can be referenced using Related Documents = field. Additional information on data dictionaries can be found here: https://www.usgs.gov/data-management/data-dictionaries | ||||||||||||||||||
| Data Dictionary Type (optional) | The machine-readable file format (IANA Media Type or MIME Type) of the distribution’s "described by" URL E.g., application/rdf+xml, text/csv, application/schema+json. | ||||||||||||||||||
| Data Standard (optional) | URI used to identify the standardized specification the dataset conforms to. | ||||||||||||||||||
| Data Quality (optional) | Description of any quality assurance/quality control procedures the dataset went through. | ||||||||||||||||||
| Distribution (optional) |
|
||||||||||||||||||
| Release Date (optional) | Date of formal issuance | ||||||||||||||||||
| Keywords |
Keywords to help users discover your dataset; please include terms that would be used by technical and non-technical users.
Examples of keywords that could be used include:
|
||||||||||||||||||
| Homepage URL (optional) |
Alternative landing page used to redirect user to a contextual. Agency-hosted “homepage” for the dataset or API
when selecting this resource from the Data.gov user interface.
For related GIS models this field can be used to capture the link to the metadata for the final GIS model. |
||||||||||||||||||
| Language (optional) | The language of the dataset (e.g., English). | ||||||||||||||||||
| License (optional) | The license dataset or API is published with. See more information about open licenses here. | ||||||||||||||||||
| Last Update | Most recent date on which the dataset was changed, updated, or modified. | ||||||||||||||||||
| Program Code (optional) | For work related to Federal agencies, list the primary program related to this data asset from the Federal Program Inventory. | ||||||||||||||||||
| Project Open Data Organization |
|
||||||||||||||||||
| Related Documents (optional) | Additional resources such as technical information about the dataset, developer documentation, etc. | ||||||||||||||||||
| Spatial (optional) |
|
||||||||||||||||||
| Temporal (optional) |
|
||||||||||||||||||
| Collection (optional) | The collection of which the dataset is a subset, if applicable. | ||||||||||||||||||
| Category (optional) | Main thematic category of the dataset. For examples, see: USGS Thesaurus categories |
FILE UPLOAD
Model files and dataset files should be uploaded in EnDMC to make them available for other users, as scoped in TO4 and as specified in the LWI Data Governance and modeling guidance methodology documents.
Each file will be associated with a specific model application (when the file is shared across multiple simulations), simulation or dataset. No files should be uploaded for software, software version or packages. To upload your model application/simulation/dataset file(s), go to the “See all” tab and identify the metadata for which you want to upload files. Open it, scroll to the bottom of the page, and drag and drop your file(s) in the “Upload Files” section. You can also select or drag and drop an entire folder. Do not upload zipped folders; as this will prevent the system and future users from seeing what is or is not within the zipped directory.
If the files shared across multiple simulations were already uploaded at the model application level, they should not be uploaded again at the simulation level (or this would create unnecessary duplication). EnDMC will include them when displaying the files for that simulation.
Please note that, depending on the software used, and on the model setting that the user choses, it might be more convenient to upload all files at the simulation level or all files at the model application level, instead of splitting them. The modeler who uploads the file(s), and likely is very familiar with the model set up, should use their best professional judgment. For example, in case of HEC-HMS and HEC-RAS models, the files are often organized in such a way that it is not convenient to separate the files used for a single simulation/plan. In that case, please upload all files at the model application level even if you are describing them in the simulation metadata. This would be most akin to how HEC models are passed from one user to another in present terms, often by having all required files in one directory. Make sure to include the right paths in the input/output file description.
If, for whatever reason, the files cannot be uploaded (e.g., they need to be saved somewhere else), make sure to include the link to them (URL link or location path) when creating the metadata under the “Model File External Location” field.
FREQUENTLY ASKED QUESTIONS
Where can I learn more about the features of EnDMC?
Overview of EnDMC functionality video (13 mins)
What browser should I use to access the catalog?
For the best experience, please use Chrome or Firefox.
Is there any file size limit?
Currently the upload file size limit is 100GB. Please do not upload zipped folders.
How do I optimize my search?
In the search bar, enter "keywords: [your search term]".
Where can I find EnDMC Desktop, which will help automate metadata creation for HEC models?
Here is a link to download EnDMC Desktop.
Do you have an API?
Yes. Please see the API documentation page. If you are an editor, you can create an access token on your account page.