Data Transformation and Submission

Should there be a brief paragraph explaining the purpose of the chapter

WIBL and YDVR data processing

Getting Set Up

You will need to install the following programs to run the WIBL processing scripts:

image.png

image.png

image.png

image.png

Perform the following steps the first time you run the WIBL scripts:

Note: Edit file paths as needed to match the location where your files are saved locally

#Open Anaconda Prompt

#Run the following code
conda init powershell

#Close Anaconda Prompt
---------------------------------------------
#Run Windows PowerShell 7 as an administrator (see screenshot below)

#Run the following code
Set-ExecutionPolicy Unrestricted

#Close Windows PowerShell 7, then reopen (not as admin)

#Set directory to the wibl-python folder within the WIBL repository
cd .\Desktop\WIBL-main\wibl-python

#Install packages
pip install .

#Create environment
conda env create -f environment.yml

#Execute the environment
conda activate wibl-python

image.png

Metadata - Configuration - Authentication

 If you did not preload metadata onto the logger before installing it, you will need to create your metadata and configuration files before you can process raw data with the WIBL scripts.

Example metadata and configuration files can be found in the WIBL repository. Guidance for how to properly format each of these file types can be found in the README file located in the same folder.

Metadata, configuration, and README file locations in the WIBL repository:
WIBL-Main\wibl-python\examples\ship-metadata-complete.json
WIBL-Main\wibl-python\examples\configure-submission.json
WIBL-Main\wibl-python\examples\configure-submission-test.json
WIBL-Main\wibl-python\examples\README,md

To create metadata and configuration files specific to your dataset:

  1. Copy the example files to a new folder in Documents (or Desktop) - don't work within the WIBL repository.
  2. Open the copied files in Visual Studio Code to edit.
  3. In the configuration file, make sure to set "local": false, or else your files will not be transmitted to the DCDB when you run submitDCDB.ps1.
  4. The configure-submission-test.json can be used to submit data to the test-API on the DCDB (does not officially submit data).
  5. Save the edited files with appropriate names (e.g., chance-maritime-metadata-complete.json and configure-submission-USF.json).

Further information on mandatory and recommended metadata can be found in IHO's Guidance to Crowdsourced Bathymetry B-12 Edition 3.0.0

The more comprehensive the metadata, the more valuable CSB products are to end-users! The DCDB will accept submissions meeting the mandatory metadata requirements, but organizations should strive to meet the highest standards recommended whenever possible.

You will need to download your organization’s Trusted Node authentication token in order to submit processed data to the DCDB. 

Running the WIBL Scripts

Note: There is a README file with more detailed information about the WIBL data management scripts and how to use them in the WIBL repository.

README file location in the WIBL repository:
WIBL-main\wibl-python\scripts\data-management\README.md

Once you're ready to start processing, perform the following steps:

#Run Windows PowerShell 7 as an administrator

#Run the following code
Set-ExecutionPolicy Unrestricted

#Close Windows PowerShell 7, then reopen (not as admin)

#Set directory to folder where raw files are saved (change folder path as needed)
cd .\Documents\CSB\Chance-Maritime\1-DataProcessing\

#Rename raw files to .wibl
Get-ChildItem * | Rename-Item -NewName { $_.Name + ".wibl" }

#Set directory to WIBL data-management scripts folder
cd .\Desktop\WIBL-main\wibl-python\scripts\data-management\

#Activate environment
conda activate wibl-python

#If you are using a Yacht Devices, Teamsurv, or other non-wibl logger, first start by using the convert to wibl script with the command below: 
./convertToWibl.ps1 [[-Source] <string>] [[-OutputFolder] <string>] [[-LogConvertPath] .\logconvert.exe]

#Process .wibl to .geojson
.\processWibl.ps1 [[-wiblPath] <string>] [[-metadataFile] <string>] [[-wiblConfig] <string>]

#Validate .geojson against csbschema
.\validateWibl.ps1 [[-inPath] <string>] [[-extension] GeoJSON]

convertToWIBL.ps1:  -Source is the file path to where your non-wibl files are saved.  -OutputFolder is the filepath to where you want your wibl files to be stored -LogConvertPath is the file path where your logconvert.exe is located. 

**Note: the source string MUST be a .zip file. IF it isn't already zipped, compress your source folder to .zip before running this step
**Note: logconvert.exe must be located in the path that you designate. its simplest if it is in the data-management folder (as that is how it is written here). If you do not have logconvert.exe, it wil throw an error. logconvert.exe can be created using the wibl infrastructure from github HERE  or emailing haley@seakeepers.org for a version of this for either Mac or PC. 

Example: .\convertToWibl.ps1 -Source .\Data\Always_Jun_2026.tgz -OutputFolder .\Data\Always_Jun_2026_Output -LogConvertPath .\logconvert.exe

processWibl.ps1: -wiblPath is the path to the folder where .wibl files are saved, -metadataFile is the path to the metadata file you created, and -wiblConfig is the path to the configuration file you created.

Example: .\processWibl.ps1 -wiblPath C:\Users\schernoch\Documents\CSB\Chance-Maritime\1-DataProcessing\ -metadataFile C:\Users\schernoch\Documents\CSB\Chance-Maritime\0-Metadata\chance-maritime-metadata-complete.json -wiblConfig C:\Users\schernoch\Documents\CSB\Chance-Maritime\0-Metadata\configure-submission-USF.json

validateWibl.ps1: -inPath is the folder where processed .geojson files are saved, -extension should be set to GeoJSON

Example: .\validateWibl.ps1 -inPath C:\Users\schernoch\Documents\CSB\Chance-Maritime\1-DataProcessing\ -extension GeoJSON

Information on the most up-to-date csbschema can be found here. Once the .geojson files have been validated against csbschema, they are ready to be submitted to the DCDB.

There is also a CSB validator tool written and hosted by the DCDB (available for download through this link) that allows CSB data to be validated by time stamp (data not collected from the future) and GPS (GPS location exists) to ensure data will actually be accepted into DCDB. This tool enables a CSB manager to validate not only that the data has something in every B-12 required section like the WIBL validator (e.g., there is a number in the “depth” category), but will actually reflect the DCDB’s data validation process and signal to practitioners if the data will be flagged in the system once submitted (e.g., the depth is 1E34, which is both beyond the transducer readable range and is in a format that cannot be utilized by DCDB, resulting in the submission being rejected).

Submission of GEOJSON data to the DCDB

All data submitted to the DCDB need to be in .geojson (preferred) or XYZ format and contain appropriate metadata fields. These requirements are outlined in DCDB guidance documents and are available in the CSB Schema on GitHub. 

Submission Recommendations

For any development or testing of workflows, the DCDB recommends running scripts without posting data to the DCDB Data Viewer by using the test endpoint. This method will ensure connectivity and returns a status from the DCDB test endpoint but will not publish the data to the CSB archive or data viewers. It allows for iterative testing as needed to establish the local workflows. Simply add /test/ to the endpoint URL, so it instead reads: https://www.ngdc.noaa.gov/ingest-external/upload/csb/test/geojson  DCDB staff are standing by to assist with any questions that arise during the testing and development process; you can use bathydata@iho.int to contact DCDB directly with questions. Submissions sent to the DCDB production endpoint are considered final submissions and are subject to a lengthy process to remove if submitted in error.

While some device loggers default to writing a certain number of rows per file (1,000 is a common example), other applications may offer exporting of data over a longer time-series (entire voyages from departure to arrival is an example if exporting from an ECDIS unit). Especially for international voyages, DCDB recommends submitting these long time-series files in "chunks" or subsetting files into daily or semi-hourly parts. In the event of an international voyage over waters within a State jurisdiction that has not approved publication of CSB data via the IHO, the entire voyage track will be masked, even if only 1 point falls within the jurisdiction of the declining State. To increase the probability of publication, subsetting data into smaller chunks is a reasonable solution. By doing so, only the smallest subset of data from the voyage would be masked in this example.

Data Submission

Once your WIBL or YDVR data have been validated, they can be submitted to the DCDB using the following PowerShell script:

#Run Windows PowerShell 7 as an administrator

#Run the following code
Set-ExecutionPolicy Unrestricted

#Close Windows PowerShell 7, then reopen (not as admin)

#Set directory to WIBL data-management scripts folder
cd .\Desktop\WIBL-main\wibl-python\scripts\data-management\

#Activate environment
conda activate wibl-python

#Submit to DCDB
.\submitDCDB.ps1 [[-inPath] <string>] [[-authFile] <string>] [[-configFile] <string>] [[-extension] *.geojson]

submitDCDB.ps1: -inPath is the folder where validated .geojson files are saved, -authFile is the authentication token .txt file, -configFile should be the same as used for processWibl.ps1, and -extension should be set to *.geojson.

Example: .\submitDCDB.ps1 -inPath C:\Users\schernoch\Documents\CSB\Chance-Maritime\2-Validated\ -authFile C:\Users\schernoch\Documents\CSB\Chance-Maritime\3-Submission\ingest-external-COMITUSF.txt -configFile C:\Users\schernoch\Documents\CSB\Chance-Maritime\0-Metadata\configure-submission-USF.json -extension *.geojson

When a file is successfully submitted to the DCDB, you should receive the following output:

image.png

Each time a file is submitted successfully, the program will print a submission ID. Make sure to log these submission IDs in case your files need to be tracked down by technical support staff at the DCDB for whatever reason.

If you are unable to batch-submit your processed data using the WIBL PowerShell script provided above, you can use either of the following options to manually submit your .geojson files to the DCDB

Option 1: Python script

Copy the following script into Visual Studio Code and replace <submissionToken_path.txt> and <geojson_path.geojson> with the appropriate file paths. Save the file as submit_geojson.py and make note of what folder you save it in.

import requests

token_path = r"<submissionToken_path.txt>"
geojson_path = r"<geojson_path.geojson>"
unique_id = "COMITUSF-e512gc2r-q565-279j-548i-e5fc6389cth2"
url = "https://www.ngdc.noaa.gov/ingest-external/upload/csb/geojson"

with open(token_path, "r") as f:
    token = f.read().strip()

headers = {
    "x-auth-token": token
}
files = {
    "file": ("wibl-raw.61.geojson", open(geojson_path, "rb"), "application/json"),
    "metadataInput": (None, f'{{"uniqueID": "{unique_id}"}}')
}

response = requests.post(url, headers=headers, files=files)

# === OUTPUT ===
print(f"Status Code: {response.status_code}")
print(response.text)

Perform the following steps to submit your .geojson file to the DCDB.

#Launch Command Prompt

#Call directory to the folder where submit_geojson.py is saved
cd .\<folder_path>

#Run the following code
python submit_geojson.py

#Record the submissionID

You should receive the following output if your file was successfully submitted to the DCDB:

image.png

Each time a file is submitted successfully, the program will print a submissionID. Make sure to log these submissionIDs in case your files need to be tracked down by technical support staff at the DCDB for whatever reason. 

If you have multiple .geojson files to submit, you will need to edit <geojson_path.geojson> within submit_geojson.py each time you submit a new file.

Options 2: Batch script

Perform the following steps to submit your .geojson file to the DCDB. Make sure to replace <submissionToken_path.txt> and <geojson_path.geojson> with the appropriate file paths.

#Copy and paste the following code (lines 3-17) into Visual Studio Code

@echo off
REM Define paths and values
set "TOKEN_FILE=<submissionToken_path.txt>"
set "GEOJSON_FILE=<geojson_path.geojson>"
set "UNIQUE_ID=COMITUSF-e512gc2r-q565-279j-548i-e5fc6389cth2"

REM Read token
set /p TOKEN=<"%TOKEN_FILE%"

REM Run curl
curl -i -X POST "https://www.ngdc.noaa.gov/ingest-external/upload/csb/geojson"^
 -H "x-auth-token: %TOKEN%"^
 -H "Content-Type: multipart/form-data"^
 -F "file=@%GEOJSON_FILE%;type=application/json"^
 -F "metadataInput={\"uniqueID\":\"%UNIQUE_ID%\"}"
 
 #Replace <submissionToken_path.txt> and <geojson_path.geojson> with appropriate file paths
 
 #Launch Command Prompt
 
 #Copy and paste your edited code into Command Prompt
 
 #Run the code
 
 #Record the submissionID

You should receive the following output if your file was successfully submitted to the DCDB:

image.png

If you have multiple .geojson files to submit, each time you submit a new file you will need to edit <geojson_path.geojson> in Visual Studio Code and then run the edited code in Command Prompt.

Graphical user interface

As of March 2025, a Graphical User Interface (GUI) is being developed to aid in the processing and submission of CSB data. The GUI should lower the barrier to entry for organizations interested in becoming involved with Seabed 2030, but lacking the time or expertise to learn the programming tools required to do so. As more information becomes available on this tool, this section will be developed more fully. This toolset can be found here.