Upload benchmarking data to S3 with Neuroglancer

[1]:
from brainlit.utils import upload
from pathlib import Path
/opt/buildhome/python3.7/lib/python3.7/site-packages/python_jsonschema_objects/__init__.py:53: UserWarning: Schema version http://json-schema.org/draft-04/schema not recognized. Some keywords and features may not be supported.
  self.schema["$schema"]

Uploading Benchmarking Images from local data locations .

This notebook demonstrates uploading the benchmarking data and associated .swc segment files. The upload destination could easily be set to a url of a cloud data server such as s3.

1) Define variables.

  • source is the root directory of the data and swc files.

    • the .tif file is in the root directory and .swc files are in a subfolder called “consensus-swcs”

  • p is the prefix string. file:// indicates a filepath, while s3:// or gc:// indicate URLs.

  • dest and dest_segments are the destinations for the uploads (in this case, filepaths).

The below paths lead to sample data in my local drive. Alter the below path definitions to point to your own local file locations.

Note:

The below upload destination points to the open-neurodata S3. Uploading data will overwrite the current benchmarking data on S3.

[2]:
source = (Path().resolve().parents[5] / "Downloads" / "validation_21").as_posix()
dest = "s3://open-neurodata/brainlit/benchmarking_data/validation_21"
dest_segments = "s3://open-neurodata/brainlit/benchmarking_data/validation_21"

2) Upload the segmentation data (.swc)

[3]:
upload.upload_segments(source, dest_segments, 1, benchmarking = True)
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-1-716b8fe203a0> in <module>
----> 1 upload.upload_segments(source, dest_segments, 1, benchmarking = True)

/opt/build/repo/brainlit/utils/upload.py in upload_segments(input_path, precomputed_path, num_mips, benchmarking)
    480     if benchmarking == True:
    481         # Getting swc scaling parameters
--> 482         f = Path(input_path).parts[4].split("_")
    483         image = f[0]
    484         date = type_to_date[image]

IndexError: tuple index out of range

3) Upload the image data (.tif)

[4]:
upload.upload_volumes(source, dest, 1, benchmarking = True)
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-1-552f14221402> in <module>
----> 1 upload.upload_volumes(source, dest, 1, benchmarking = True)

/opt/build/repo/brainlit/utils/upload.py in upload_volumes(input_path, precomputed_path, num_mips, parallel, chosen, benchmarking, continue_upload)
    328     if benchmarking == True:
    329         (files_ordered, bin_paths, vox_size, img_size, _) = get_volume_info(
--> 330             input_path, num_mips, benchmarking=True
    331         )
    332         vols = create_cloud_volume(

/opt/build/repo/brainlit/utils/upload.py in get_volume_info(image_dir, num_resolutions, channel, extension, benchmarking)
     96
     97     if benchmarking == True:
---> 98         img_size = np.squeeze(tf.imread(str(p / files[0][parent_dirs]))).T.shape
     99         vox_size = img_size
    100

IndexError: list index out of range

Appendix

  • If when downloading, you get a reshape error, try uploading the segments before uploading the volumes

[5]:
test_list = []
validation_list = []
for i in range(25):
    test_list.append('test_' + str(i+1))
    validation_list.append('validation_' + str(i+1))
[6]:
num_res = 1

for test in test_list:
    print(test)
    source = (Path().resolve().parents[5] / "Downloads" / test).as_posix()
    dest = "s3://open-neurodata/brainlit/benchmarking_data/" + test
    dest_segments = "s3://open-neurodata/brainlit/benchmarking_data/" + test
    upload.upload_segments(source, dest_segments, num_res, benchmarking = True)
    upload.upload_volumes(source, dest, num_res, benchmarking = True)

for val in validation_list:
    print(val)
    source = (Path().resolve().parents[5] / "Downloads" / val).as_posix()
    dest = "s3://open-neurodata/brainlit/benchmarking_data/" + val
    dest_segments = "s3://open-neurodata/brainlit/benchmarking_data/" + val
    upload.upload_segments(source, dest_segments, num_res, benchmarking = True)
    upload.upload_volumes(source, dest, num_res, benchmarking = True)
test_1
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-1-503aad61967a> in <module>
      6     dest = "s3://open-neurodata/brainlit/benchmarking_data/" + test
      7     dest_segments = "s3://open-neurodata/brainlit/benchmarking_data/" + test
----> 8     upload.upload_segments(source, dest_segments, num_res, benchmarking = True)
      9     upload.upload_volumes(source, dest, num_res, benchmarking = True)
     10

/opt/build/repo/brainlit/utils/upload.py in upload_segments(input_path, precomputed_path, num_mips, benchmarking)
    480     if benchmarking == True:
    481         # Getting swc scaling parameters
--> 482         f = Path(input_path).parts[4].split("_")
    483         image = f[0]
    484         date = type_to_date[image]

IndexError: tuple index out of range