Skip to main content

Submission of GEOJSON data to the DCDB

All data submitted to the DCDB need to be in .geojson (preferred) or XYZ format and contain appropriate metadata fields. These requirements are outlined in DCDB guidance documents and are available in the CSB Schema on GitHub. 

Submission Recommendations

For any development or testing of workflows, the DCDB recommends running scripts without posting data to the DCDB Data Viewer by using the test endpoint. This method will ensure connectivity and returns a status from the DCDB test endpoint but will not publish the data to the CSB archive or data viewers. It allows for iterative testing as needed to establish the local workflows. Simply add /test/ to the endpoint URL, so it instead reads: https://www.ngdc.noaa.gov/ingest-external/upload/csb/test/geojson  DCDB staff are standing by to assist with any questions that arise during the testing and development process; you can use bathydata@iho.int to contact DCDB directly with questions. Submissions sent to the DCDB production endpoint are considered final submissions and are subject to a lengthy process to remove if submitted in error.

While some device loggers default to writing a certain number of rows per file (1,000 is a common example), other applications may offer exporting of data over a longer time-series (entire voyages from departure to arrival is an example if exporting from an ECDIS unit). Especially for international voyages, DCDB recommends submitting these long time-series files in "chunks" or subsetting files into daily or semi-hourly parts. In the event of an international voyage over waters within a State jurisdiction that has not approved publication of CSB data via the IHO, the entire voyage track will be masked, even if only 1 point falls within the jurisdiction of the declining State. To increase the probability of publication, subsetting data into smaller chunks is a reasonable solution. By doing so, only the smallest subset of data from the voyage would be masked in this example.

Data Submission

  • For WIBL or YDVR data, you will need to upload data periodically. See the WIBL and YDVR Data Processing protocol for full details.
  • For Mussel Kit data, OFM handles data submission on behalf of the Trusted Node, using the Trusted Node's secure token.
  • OFM can also provide the raw data back to Trusted Nodes via Dropbox or AWS S3 buckets (other options are also available).

Once your WIBL or YDVR data have been validated, they can be submitted to the DCDB using the following PowerShell script:

#Run Windows PowerShell 7 as an administrator

#Run the following code
Set-ExecutionPolicy Unrestricted

#Close Windows PowerShell 7, then reopen (not as admin)

#Set directory to WIBL data-management scripts folder
cd .\Desktop\WIBL-main\wibl-python\scripts\data-management\

#Activate environment
conda activate wibl-python

#Submit to DCDB
.\submitDCDB.ps1 [[-inPath] <string>] [[-authFile] <string>] [[-configFile] <string>] [[-extension] *.geojson]

submitDCDB.ps1: -inPath is the folder where validated .geojson files are saved, -authFile is the authentication token .txt file, -configFile should be the same as used for processWibl.ps1, and -extension should be set to *.geojson.

Example: .\submitDCDB.ps1 -inPath C:\Users\schernoch\Documents\CSB\Chance-Maritime\2-Validated\ -authFile C:\Users\schernoch\Documents\CSB\Chance-Maritime\3-Submission\ingest-external-COMITUSF.txt -configFile C:\Users\schernoch\Documents\CSB\Chance-Maritime\0-Metadata\configure-submission-USF.json -extension *.geojson

When a file is successfully submitted to the DCDB, you should receive the following output:

image.png

Each time a file is submitted successfully, the program will print a submission ID. Make sure to log these submission IDs in case your files need to be tracked down by technical support staff at the DCDB for whatever reason.

If you are unable to batch-submit your processed data using the WIBL PowerShell script provided above, you can use either of the following options to manually submit your .geojson files to the DCDB

Option 1: Python script

Copy the following script into Visual Studio Code and replace <submissionToken_path.txt> and <geojson_path.geojson> with the appropriate file paths. Save the file as submit_geojson.py and make note of what folder you save it in.

import requests

token_path = r"<submissionToken_path.txt>"
geojson_path = r"<geojson_path.geojson>"
unique_id = "COMITUSF-e512gc2r-q565-279j-548i-e5fc6389cth2"
url = "https://www.ngdc.noaa.gov/ingest-external/upload/csb/geojson"

with open(token_path, "r") as f:
    token = f.read().strip()

headers = {
    "x-auth-token": token
}
files = {
    "file": ("wibl-raw.61.geojson", open(geojson_path, "rb"), "application/json"),
    "metadataInput": (None, f'{{"uniqueID": "{unique_id}"}}')
}

response = requests.post(url, headers=headers, files=files)

# === OUTPUT ===
print(f"Status Code: {response.status_code}")
print(response.text)

Perform the following steps to submit your .geojson file to the DCDB.

#Launch Command Prompt

#Call directory to the folder where submit_geojson.py is saved
cd .\<folder_path>

#Run the following code
python submit_geojson.py

#Record the submissionID

You should receive the following output if your file was successfully submitted to the DCDB:

image.png

Each time a file is submitted successfully, the program will print a submissionID. Make sure to log these submissionIDs in case your files need to be tracked down by technical support staff at the DCDB for whatever reason. 

If you have multiple .geojson files to submit, you will need to edit <geojson_path.geojson> within submit_geojson.py each time you submit a new file.

  • This can be done relatively quickly if you keep submit_geojson.py open within Visual Studio Code, making sure to save (ctrl+S) your changes before running the code again.
  • Once <geojson_path.geojson> has been edited to the new file path, you can click the up arrow in Command Prompt to reload the previous command and click enter to run it.

Options 2: Batch script

Perform the following steps to submit your .geojson file to the DCDB. Make sure to replace <submissionToken_path.txt> and <geojson_path.geojson> with the appropriate file paths.

#Copy and paste the following code (lines 3-17) into Visual Studio Code

@echo off
REM Define paths and values
set "TOKEN_FILE=<submissionToken_path.txt>"
set "GEOJSON_FILE=<geojson_path.geojson>"
set "UNIQUE_ID=COMITUSF-e512gc2r-q565-279j-548i-e5fc6389cth2"

REM Read token
set /p TOKEN=<"%TOKEN_FILE%"

REM Run curl
curl -i -X POST "https://www.ngdc.noaa.gov/ingest-external/upload/csb/geojson"^
 -H "x-auth-token: %TOKEN%"^
 -H "Content-Type: multipart/form-data"^
 -F "file=@%GEOJSON_FILE%;type=application/json"^
 -F "metadataInput={\"uniqueID\":\"%UNIQUE_ID%\"}"
 
 #Replace <submissionToken_path.txt> and <geojson_path.geojson> with appropriate file paths
 
 #Launch Command Prompt
 
 #Copy and paste your edited code into Command Prompt
 
 #Run the code
 
 #Record the submissionID

You should receive the following output if your file was successfully submitted to the DCDB:

image.png

If you have multiple .geojson files to submit, each time you submit a new file you will need to edit <geojson_path.geojson> in Visual Studio Code and then run the edited code in Command Prompt.

No comments to display

Back to top