Upload benchmarking data to S3 with Neuroglancer

from brainlit.utils import upload
from pathlib import Path

Uploading Benchmarking Images from local data locations .

This notebook demonstrates uploading the benchmarking data and associated .swc segment files. The upload destination could easily be set to a url of a cloud data server such as s3.

1) Define variables.

• source is the root directory of the data and swc files.

  • the .tif file is in the root directory and .swc files are in a subfolder called “consensus-swcs”

• p is the prefix string. file:// indicates a filepath, while s3:// or gc:// indicate URLs.

• dest and dest_segments are the destinations for the uploads (in this case, filepaths).

The below paths lead to sample data in my local drive. Alter the below path definitions to point to your own local file locations.

Note:

The below upload destination points to the open-neurodata S3. Uploading data will overwrite the current benchmarking data on S3.

source = (Path().resolve().parents[5] / "Downloads" / "validation_21").as_posix()
dest = "s3://open-neurodata/brainlit/benchmarking_data/validation_21"
dest_segments = "s3://open-neurodata/brainlit/benchmarking_data/validation_21"

2) Upload the segmentation data (.swc)

upload.upload_segments(source, dest_segments, 1, benchmarking=True)

3) Upload the image data (.tif)

upload.upload_volumes(source, dest, 1, benchmarking=True)

Appendix

• If when downloading, you get a reshape error, try uploading the segments before uploading the volumes

test_list = []
validation_list = []
for i in range(25):
    test_list.append("test_" + str(i + 1))
    validation_list.append("validation_" + str(i + 1))

num_res = 1

for test in test_list:
    print(test)
    source = (Path().resolve().parents[5] / "Downloads" / test).as_posix()
    dest = "s3://open-neurodata/brainlit/benchmarking_data/" + test
    dest_segments = "s3://open-neurodata/brainlit/benchmarking_data/" + test
    upload.upload_segments(source, dest_segments, num_res, benchmarking=True)
    upload.upload_volumes(source, dest, num_res, benchmarking=True)

for val in validation_list:
    print(val)
    source = (Path().resolve().parents[5] / "Downloads" / val).as_posix()
    dest = "s3://open-neurodata/brainlit/benchmarking_data/" + val
    dest_segments = "s3://open-neurodata/brainlit/benchmarking_data/" + val
    upload.upload_segments(source, dest_segments, num_res, benchmarking=True)
    upload.upload_volumes(source, dest, num_res, benchmarking=True)