Immersive – Jaffe Laboratory for Underwater Imaging

VRMacroscope

Sponsors: National Science Foundation

Project Overview:

The VRMacroscopic is a custom built opto-mechnical system that can capture 360 degree stereoscopic panoramic timelapse videos of microscopic scenes and generate stitched frames in realtime. When the content is viewed in a virtual reality environment (headset or immersive theatre) the effect experienced by the viewer is that of becoming a miniature explorer, approximately 5mm in size, and placed in the center of the microscopic scene. Since the system records time-lapse videos the viewer can explore typically invisible phenomena through novel manipulations of both time and space.

2022

Jaffe, Jules S.

Non-invasive estimation of coral polyp volume and surface area using optical coherence tomography Journal Article

In: Frontiers in Marine Science, vol. 9, 2022, ISSN: 2296-7745.

Abstract | Links | BibTeX

@article{Jaffe2022,

title = {Non-invasive estimation of coral polyp volume and surface area using optical coherence tomography},

author = {Jaffe, Jules S.},

editor = {Jian Sheng},

url = {https://doi.org/10.3389/fmars.2022.1049440},

doi = {10.3389/fmars.2022.1049440},

issn = {2296-7745},

year  = {2022},

date = {2022-12-12},

urldate = {2022-12-12},

journal = {Frontiers in Marine Science},

volume = {9},

abstract = {The surface area (SA) and three-dimensional (3D) morphology of reef-building corals are central to their physiology. A challenge for the estimation of coral SA has been to meet the required spatial resolution as well as the capability to preserve the soft tissue in its native state during measurements. Optical Coherence Tomography (OCT) has been used to quantify the 3D microstructure of coral tissues and skeletons with nearly micron-scale resolution. Here, we develop a non-invasive method to quantify surface area and volume of single coral polyps. A coral fragment with several coral polyps as well as calibration targets of known areal extent are scanned with an OCT system. This produces a 3D matrix of optical backscatter that is analyzed with computer algorithms to detect refractive index mismatches between physical boundaries between the coral and the immersed water. The algorithms make use of a normalization of the depth dependent scatter intensity and signal attenuation as well as region filling to depict the interface between the coral soft tissue and the water. Feasibility of results is judged by inspection as well as by applying algorithms to hard spheres and fish eggs whose volume and SA can be estimated analytically. The method produces surface area estimates in calibrated targets that are consistent with analytic estimates within 93%. The appearance of the coral polyp surfaces is consistent with visual inspection that permits standard programs to visualize both point clouds and 3-D meshes. The method produces the 3-D definition of coral tissue and skeleton at a resolution close to 10 µm, enabling robust quantification of polyp volume to surface area ratios.},

keywords = {},

pubstate = {published},

tppubtype = {article}

}

Kevin T. Le, et al

Benchmarking and Automating the Image Recognition Capability of an In Situ Plankton Imaging System Journal Article

In: Frontiers in Marine Science, vol. 9, 2022, ISBN: 2296-7745.

Abstract | Links | BibTeX

@article{Le2022b,

title = {Benchmarking and Automating the Image Recognition Capability of an In Situ Plankton Imaging System},

author = {Le, Kevin T., et al},

editor = {Mark C. Benfield},

url = {https://www.frontiersin.org/article/10.3389/fmars.2022.869088},

doi = {10.3389/fmars.2022.869088},

isbn = {2296-7745},

year  = {2022},

date = {2022-06-10},

urldate = {2022-06-10},

journal = {Frontiers in Marine Science},

volume = {9},

abstract = {To understand ocean health, it is crucial to monitor photosynthetic marine plankton – the microorganisms that form the base of the marine food web and are responsible for the uptake of atmospheric carbon. With the recent development of in situ microscopes that can acquire vast numbers of images of these organisms, the use of deep learning methods to taxonomically identify them has come to the forefront. Given this, two questions arise: 1) How well do deep learning methods such as Convolutional Neural Networks (CNNs) identify these marine organisms using data from in situ microscopes? 2) How well do CNN-derived estimates of abundance agree with established net and bottle-based sampling? Here, using images collected by the in situ Scripps Plankton Camera (SPC) system, we trained a CNN to recognize 9 species of phytoplankton, some of which are associated with Harmful Algal Blooms (HABs). The CNNs evaluated on 26 independent natural samples collected at Scripps Pier achieved an averaged accuracy of 92%, with 7 of 10 target categories above 85%. To compare abundance estimates, we fit a linear model between the number of organisms of each species counted in a known volume in the lab, with the number of organisms collected by the in situ microscope sampling at the same time. The linear fit between lab and in situ counts of several of the most abundant key HAB species suggests that, in the case of dinoflagellates, there is good correspondence between the two methods. As one advantage of our method, given the excellent correlation between lab counts and in situ microscope counts for key species, the methodology proposed here provides a way to estimate an equivalent volume in which the employed microscope can identify in-focus organisms and obtain statistically robust estimates of abundance.},

keywords = {},

pubstate = {published},

tppubtype = {article}

}

Lindemann Yoav Ben-Zvi Or, Eyal Gal

Coral fluorescence: a prey-lure in deep habitats Journal Article

In: Communications Biology, vol. 5, iss. 1, no. 5, 2022, ISSN: 2399-3642.

Abstract | Links | BibTeX

Pichaya Lertvilai, Jules S. Jaffe

In situ size and motility measurement of aquatic invertebrates with an underwater stereoscopic camera system using tilted lenses Journal Article

In: Methods in Ecology and Evolution, 2022.

Abstract | Links | BibTeX

@article{Lertvilai2022,

title = {In situ size and motility measurement of aquatic invertebrates with an underwater stereoscopic camera system using tilted lenses},

author = {Pichaya Lertvilai, Jules S. Jaffe},

editor = {Marta Vidal Garcia},

url = {https://besjournals.onlinelibrary.wiley.com/doi/abs/10.1111/2041-210X.13855},

doi = {https://doi.org/10.1111/2041-210X.13855},

year  = {2022},

date = {2022-03-28},

urldate = {2022-03-28},

journal = {Methods in Ecology and Evolution},

abstract = {Abstract In situ observation of traits of aquatic organisms, including size and motility, requires three-dimensional measurements that are commonly done with a stereoscopic imaging system. However, to observe traits of small aquatic invertebrates, the imaging system requires relatively high magnification, which results in a small overlapping volume between the two cameras of a conventional stereoscopic system. The provision of a larger shared volume would therefore be of great advantage, especially, when the organism abundance is low. We implement a stereoscopic system that utilizes a tilted lens approach, known as the Scheimpflug principle, to increase the common imaging volume of two cameras. The system was calibrated and tested in the laboratory and then deployed in a saltmarsh to observe water boatmen Trichocorixa californica. Processing of the image data from the field deployments resulted in the simultaneous estimation of the traits of body length and swimming speed of the aquatic insects. Our stereo setup with tilted lenses increased the sampling volume by 3.1 times compared to a traditional stereo setup with the same optical parameters. The in situ data and subsequent processing reveal that the instrument can capture stereoscopic images that resolve both body length and swimming speed of the aquatic insects. Results indicate that the relationship between the body length and the swimming speed of the water boatmen is linear in the log–log space with an exponent of 0.81±0.12$$ 0.81pm 0.12 $$. Furthermore, the insects experience Reynold's number in the range of 100$$ {10}^0 $$–103$$ {10}^3 $$. Our results demonstrated that the system can be used to observe key traits of small aquatic organisms in an ecologically relevant context. This work expands the capability of underwater imaging systems to measure important traits of an individual aquatic invertebrate in its natural environment and aids in providing a trait-based approach to zooplankton ecology.},

keywords = {},

pubstate = {published},

tppubtype = {article}

}

Jon S. Sauer, et al

The Sea Spray Chemistry and Particle Evolution study (SeaSCAPE): overview and experimental methods Journal Article

In: Environmental Science: Processes and Impacts, 2022.

2021

Ewa Merz, et al

Underwater dual-magnification imaging for automated lake plankton monitoring Journal Article

In: Water Research, vol. 203, no. 117524, 2021.

Abstract | Links | BibTeX

@article{Merz2021,

title = {Underwater dual-magnification imaging for automated lake plankton monitoring},

author = {Ewa Merz, et al},

url = {https://doi.org/10.1016/j.watres.2021.117524},

year  = {2021},

date = {2021-09-15},

urldate = {2021-09-15},

journal = {Water Research},

volume = {203},

number = {117524},

abstract = {The Dual Scripps Plankton Camera (DSPC) is a new approach for automated in-situ monitoring of phyto- and zooplankton communities based on a dual magnification dark-field imaging microscope. Here, we present the DSPC and its associated image processing while evaluating its capabilities in i) detecting and characterizing plankton species of different size and taxonomic categories and ii) measuring their abundance in both laboratory and field applications. In the laboratory, body size and abundance estimates by the DSPC significantly and robustly scaled with measurements derived by microscopy. In the field, a DSPC installed permanently at 3 m depth in Lake Greifensee (Switzerland) delivered images of plankton individuals, colonies, and heterospecific aggregates at hourly timescales without disrupting natural arrangements of interacting organisms, their microenvironment or their behavior. The DSPC was able to track the dynamics of taxa, mostly at the genus level, in the size range between ∼10 μm to ∼ 1 cm, covering many components of the planktonic food web (including parasites and potentially toxic cyanobacteria). Comparing data from the field-deployed DSPC to traditional sampling and microscopy revealed a general overall agreement in estimates of plankton diversity and abundances. The most significant disagreements between traditional methods and the DSPC resided in the measurements of zooplankton community properties. Our data suggest that the DSPC is better equipped to study the dynamics and demography of heterogeneously distributed organisms such as zooplankton, because high temporal resolution and continuous sampling offer more information and less variability in taxa detection and quantification than traditional sampling. Time series collected by the DSPC depicted ecological succession patterns, algal bloom dynamics and diel fluctuations with a temporal frequency and morphological resolution that was never observed by traditional methods. Access to high frequency, reproducible and real-time data of a large spectrum of the planktonic ecosystem expands our understanding of both applied and fundamental plankton ecology. We conclude the DSPC is robust for both research and water quality monitoring and suitable for stable long-term deployments.},

keywords = {},

pubstate = {published},

tppubtype = {article}

}

The Dual Scripps Plankton Camera (DSPC) is a new approach for automated in-situ monitoring of phyto- and zooplankton communities based on a dual magnification dark-field imaging microscope. Here, we present the DSPC and its associated image processing while evaluating its capabilities in i) detecting and characterizing plankton species of different size and taxonomic categories and ii) measuring their abundance in both laboratory and field applications. In the laboratory, body size and abundance estimates by the DSPC significantly and robustly scaled with measurements derived by microscopy. In the field, a DSPC installed permanently at 3 m depth in Lake Greifensee (Switzerland) delivered images of plankton individuals, colonies, and heterospecific aggregates at hourly timescales without disrupting natural arrangements of interacting organisms, their microenvironment or their behavior. The DSPC was able to track the dynamics of taxa, mostly at the genus level, in the size range between ∼10 μm to ∼ 1 cm, covering many components of the planktonic food web (including parasites and potentially toxic cyanobacteria). Comparing data from the field-deployed DSPC to traditional sampling and microscopy revealed a general overall agreement in estimates of plankton diversity and abundances. The most significant disagreements between traditional methods and the DSPC resided in the measurements of zooplankton community properties. Our data suggest that the DSPC is better equipped to study the dynamics and demography of heterogeneously distributed organisms such as zooplankton, because high temporal resolution and continuous sampling offer more information and less variability in taxa detection and quantification than traditional sampling. Time series collected by the DSPC depicted ecological succession patterns, algal bloom dynamics and diel fluctuations with a temporal frequency and morphological resolution that was never observed by traditional methods. Access to high frequency, reproducible and real-time data of a large spectrum of the planktonic ecosystem expands our understanding of both applied and fundamental plankton ecology. We conclude the DSPC is robust for both research and water quality monitoring and suitable for stable long-term deployments.

Ronen, Roi; Attias, Yacov; Schechner, Yoay Y.; Jaffe, Jules S.; Orenstein, Eric C.

Plankton reconstruction through robust statistical optical tomography Journal Article

In: Journal of the Optical Society of America A, vol. 38, no. 9, pp. 1320-1331, 2021.

Abstract | Links | BibTeX

Camille M. L. S. Pagniello, et al

An Optical Imaging System for Capturing Images in Low-Light Aquatic Habitats Using Only Ambient Light Journal Article

In: Oceanography Magazine, 2021.

Abstract | Links | BibTeX

Lertvilai, Pichaya; Roberts, Paul L. D.; Jaffe, Jules S.

In Situ Underwater Average Flow Velocity Estimation Using a Low-Cost Video Velocimeter Journal Article

In: Journal of Atmospheric and Oceanic Technology, vol. 38, no. 6, pp. 1143-1156, 2021.

Abstract | Links | BibTeX

2020

Garwood, Jessica C.; Lucas, Andrew J.; Naughton, Perry; Roberts, Paul L. D.; Jaffe, Jules S; deGelleke, Laura; Franks, Peter J. S.

Larval cross-shore transport estimated from internal waves with a mean flow: the effects of larval vertical position and depth regulation. Journal Article

In: Limnology and Oceanography, 2020.

Abstract | Links | BibTeX

@article{Garwood2020b,

title = {Larval cross-shore transport estimated from internal waves with a mean flow: the effects of larval vertical position and depth regulation.},

author = {Jessica C. Garwood and Andrew J. Lucas and Perry Naughton and Paul L. D. Roberts and Jules S Jaffe and Laura deGelleke and Peter J. S. Franks},

url = {https://doi.org/10.1002/lno.11632},

year  = {2020},

date = {2020-10-26},

journal = {Limnology and Oceanography},

abstract = {Cross‐shore velocities in the coastal ocean typically vary with depth. The direction and magnitude of transport experienced by meroplanktonic larvae will therefore be influenced by their vertical position. To quantify how swimming behavior and vertical position in internal waves influence larval cross‐shore transport in the shallow (~ 20 m), stratified coastal waters off Southern California, we deployed swarms of novel, subsurface larval mimics, the Mini‐Autonomous Underwater Explorers (M‐AUEs). The M‐AUEs were programmed to maintain a specified depth, and were deployed near a mooring. Transport of the M‐AUEs was predominantly onshore, with average velocities up to 14 cm s−1. To put the M‐AUE deployments into a broader context, we simulated > 500 individual high‐frequency internal waves observed at the mooring over a 14‐d deployment; in each internal wave, we released both depth‐keeping and passive virtual larvae every meter in the vertical. After the waves' passage, depth‐keeping virtual larvae were usually found closer to shore than passive larvae released at the same depth. Near the top of the water column (3–5‐m depth), ~ 20% of internal waves enhanced onshore transport of depth‐keeping virtual larvae by ≥ 50 m, whereas only 1% of waves gave similar enhancements to passive larvae. Our observations and simulations showed that depth‐keeping behavior in high‐frequency internal waves resulted in enhanced onshore transport at the top of the water column, and reduced offshore dispersal at the bottom, compared to being passive. Thus, even weak depth‐keeping may allow larvae to reach nearshore adult habitats more reliably than drifting passively.},

keywords = {},

pubstate = {published},

tppubtype = {article}

}

Orenstein, Eric C.; Ratelle, Devin; Briseño-Avena, Christian; Carter, Melissa L; Franks, Peter J. S.; Jaffe, Jules S.; Roberts, Paul L. D.

The Scripps Plankton Camera system: A framework and platform for in situ microscopy Journal Article

In: Limnology and Oceanography: Methods, 2020.

Abstract | Links | BibTeX

Briseño-Avena, Christian; Prairie, Jennifer C.; Franks, Peter J. S.; Jaffe, Jules S.

Comparing Vertical Distributions of Chl-a Fluorescence, Marine Snow, and Taxon-Specific Zooplankton in Relation to Density Using High Resolution Optical Measurements Journal Article

In: Frontiers in Marine Science, 2020.

Abstract | Links | BibTeX

@article{Briseño-Avena2020,

title = {Comparing Vertical Distributions of Chl-a Fluorescence, Marine Snow, and Taxon-Specific Zooplankton in Relation to Density Using High Resolution Optical Measurements},

author = {Christian Briseño-Avena and Jennifer C. Prairie and Peter J. S. Franks and Jules S. Jaffe},

url = {https://www.frontiersin.org/articles/10.3389/fmars.2020.00602/abstract},

doi = {10.3389/fmars.2020.00602},

year  = {2020},

date = {2020-07-28},

journal = {Frontiers in Marine Science},

abstract = {Interactions between predators and their prey are important in shaping planktonic ecosystems. However, these interactions are difficult to assess in situ at the spatial scales relevant to the organisms. This work presents high spatial resolution observations of the nighttime vertical distributions of individual zooplankton, chlorophyll-a fluorescence, and marine snow in stratified coastal waters of the Southern California Bight. Data were obtained using a planar laser imaging fluorometer (PLIF) augmented with a shadowgraph zooplankton imaging system (O-Cam) mounted along with ancillary sensors on a free-descent platform. Fluorometer and PLIF sensors detected two well-defined and distinct peaks: the subsurface chlorophyll maximum (SCM) and a fluorescent particle maximum (FPM) dominated by large marine snow. The O-Cam imaging system allows reliable estimates of concentrations of crustacean and gelatinous zooplankton groups; we found that grazers and their predators had well-structured nighttime distributions in and around the SCM and FPM in ways that suggested potential predator avoidance at the peak of the SCM and immediately above the FPM (where predatory hydromedusae, and to some degree euphausiids, were primarily located). Calanoid copepods were found above the SCM while cyclopoids were associated with the FPM. The locations of predator and grazer concentration peaks suggest that their dynamics may control the vertical gradients defining the SCM and FPM.},

keywords = {},

pubstate = {published},

tppubtype = {article}

}

Lertvilai, Pichaya

The In situ Plankton Assemblage eXplorer (IPAX): An inexpensive underwater imaging system for zooplankton study Journal Article

In: Methods in Ecology and Evolution, 2020.

Abstract | Links | BibTeX

@article{Lertvilai2020,

title = {The In situ Plankton Assemblage eXplorer (IPAX): An inexpensive underwater imaging system for zooplankton study},

author = {Pichaya Lertvilai},

url = {https://doi.org/10.1111/2041-210X.13441},

year  = {2020},

date = {2020-06-27},

journal = {Methods in Ecology and Evolution},

abstract = {1. Zooplankton play vital ecological roles that maintain aquatic ecosystems. Imaging instruments have enabled in situ observations of these organisms that can be automated and are less invasive than traditional sampling methods. However, these instruments are often costly and require sophisticated engineering expertise to operate.

2. The In situ Plankton Assemblage eXplorer (IPAX) is an open‐source low‐cost imaging platform for zooplankton studies. The IPAX is a programmable instrument that has powerful LED illumination and a high‐resolution camera that can image zooplankton in situ, while material costs are less than USD $450. The optical performance of the instrument was calibrated in the laboratory using a calibration target and preserved zooplankton. The IPAX was then deployed in the field to observe diversity, emergent patterns and phototactic behaviour of demersal zooplankton at night to demonstrate its practicality.

3. Laboratory calibration indicated that the IPAX can resolve 100 µm features with 70% contrast at the focal plane with 5 cm × 3 cm field of view and 5 mm depth of field. The instrument also resolved fine morphological details of preserved zooplankton when in focus. The field deployment demonstrated capability to resolve the myriad of zooplankton present in addition to the different phototactic behaviour that was elicited and observed from the different colour LEDs.

4. The IPAX enables economical and autonomous surveys of zooplankton in various aquatic habitats. Its low cost facilitates construction and deployment of multiple units that can cover large spatial areas, while its versatility also allows adaptations to many experimental needs for aquatic ecology.},

keywords = {},

pubstate = {published},

tppubtype = {article}

}

Kenitz, Kasia M; Orenstein, Eric C.; Roberts, Paul L D; Franks, Peter J S; Jaffe, Jules S; Carter, Melissa L; Barton, Andrew D

Environmental drivers of population variability in colony‐forming marine diatoms Journal Article

In: Limnology and Oceanography, 2020.

Abstract | Links | BibTeX

@article{Kenitz2020,

title = {Environmental drivers of population variability in colony‐forming marine diatoms},

author = {Kasia M Kenitz and Eric C. Orenstein and Paul L D Roberts and Peter J S Franks and Jules S Jaffe and Melissa L Carter and Andrew D Barton},

editor = {Ilana Berman-Frank},

url = {https://doi.org/10.1002/lno.11468},

year  = {2020},

date = {2020-05-26},

journal = {Limnology and Oceanography},

abstract = {Many aquatic microbes form colonies, yet little is known about their abundance and fitness relative to single‐celled taxa. The formation of diatom chains, in particular, has implications for diatom growth, survival, and carbon transfer. Here, we utilize an autonomous underwater microscope, combined with traditional microscopy, to develop a novel, multiyear record of the abundance of single‐cell and colony‐forming diatoms at Scripps Pier, a coastal location in the Southern California Bight. The total abundance of diatoms was lower during the warmer and more stratified conditions from 2015 to early 2016, but increased in cooler and less stratified conditions in mid‐2016 to late 2017. Diatom blooms were dominated by chain‐forming taxa, whereas solitary diatoms prevailed during low‐biomass conditions. The abundance of dinoflagellates, some of which are important diatom predators, is highest when colonies (chains) are most abundant. These observations of the diatom assemblage are consistent with a trade‐off between resource acquisition and predator defenses. Solitary diatom cells dominated during conditions with weak nutrient supply because they have a greater diffusive catchment area per cell in comparison to cells living in colonies. In contrast, during bloom conditions when nutrient supply is high and predators are abundant, forming a colony may reduce predation losses to quickly growing microzooplankton predators, and afford chains a higher fitness despite the costs of sharing resources with neighboring cells. These results highlight the contrasting ecology of single‐cell and chain‐forming diatoms, and the need to differentiate them in monitoring campaigns and ecological models.},

keywords = {},

pubstate = {published},

tppubtype = {article}

}

Campbell, Robert W; Roberts, Paul L D; Jaffe, Jules S

The Prince William Sound Plankton Camera: a profiling in situ observatory of plankton and particulates Journal Article

In: ICES Journal of Marine Science, vol. 77, no. 4, pp. 1440-1455, 2020.

Abstract | Links | BibTeX

Garwood, Jessica C.; Lucas, Andrew J.; Naughton, Perry; Alford, Matthew H.; Roberts, Paul L. D.; Jaffe, Jules S.; Franks, Peter J. S.

A novel cross‐shore transport mechanism revealed by subsurface, robotic larval mimics: Internal wave deformation of the background velocity field Journal Article

In: Limnology and Oceanography, vol. 65, no. 7, pp. 1456-1470, 2020.

Abstract | Links | BibTeX

@article{Garwood2020,

title = {A novel cross‐shore transport mechanism revealed by subsurface, robotic larval mimics: Internal wave deformation of the background velocity field},

author = {Jessica C. Garwood and Andrew J. Lucas and Perry Naughton and Matthew H. Alford and Paul L. D. Roberts and Jules S. Jaffe and Peter J. S. Franks},

editor = {Julia Mullarney},

url = {https://doi.org/10.1002/lno.11400},

year  = {2020},

date = {2020-01-13},

journal = {Limnology and Oceanography},

volume = {65},

number = {7},

pages = {1456-1470},

abstract = {Coastal physical processes are essential for the cross‐shore transport of meroplanktonic larvae to their benthic adult habitats. To investigate these processes, we released a swarm of novel, trackable, subsurface vehicles, the Mini‐Autonomous Underwater Explorers (M‐AUEs), which we programmed to mimic larval depth‐keeping behavior. The M‐AUE swarm measured a sudden net onshore transport of 30–70 m over 15–20 min, which we investigated in detail. Here, we describe a novel transport mechanism of depth‐keeping plankton revealed by these observations. In situ measurements and models showed that, as a weakly nonlinear internal wave propagated through the swarm, it deformed surface‐intensified, along‐isopycnal background velocities downward, accelerating depth‐keeping organisms onshore. These higher velocities increased both the depth‐keepers' residence time in the wave and total cross‐shore displacement, leading to wave‐induced transports twice those of fully Lagrangian organisms and four times those associated with the unperturbed background currents. Our analyses also show that integrating velocity time series from virtual larvae or mimics moving with the flow yields both larger and more accurate transport estimates than integrating velocity time series obtained at a point (Eulerian). The increased cross‐shore transport of organisms capable of vertical swimming in this wave/background‐current system is mathematically analogous to the increase in onshore transport associated with horizontal swimming in highly nonlinear internal waves. However, the mechanism described here requires much weaker swimming speeds (mm s−1 vs. cm s−1) to achieve significant onshore transports, and meroplanktonic larvae only need to orient themselves vertically, not horizontally.},

keywords = {},

pubstate = {published},

tppubtype = {article}

}