menu
.png)
By Louise Carver
She will be alone until she finds the others. A world of swirl and spin in an infinite expanse. Eddies thrust around her juvenile shell; a mere spec in the delicious whirl of the sparkling water known as the Atlantic at a distance beyond comprehension from the shoreline that is known as Portugal. As a small one, and at this stage of her life, the churn is intense. She has little sense of anything more than movement within this spiral and circulation through the flickering expanse of the total surround, suspended in movement. It is tantalising uncertainty and risk. All she intuits is the search and she will wait until it is time to receive.
Smelling the others are near, and also the closeness of others not at all like her, but with watchful eyes entrained on and encircling her, she senses a glimmer of opportunity. Making contact with her single foot, she extends the tendrils of her filamented limbs, her byssal threads, to first locate then grip and settle at a fixed point on the substrate; bringing, at last, some stillness. Movement will be minimised now to a gentle sway. She is tethered to an earthly ground. The rhythm is synchronised and structured by the crowded mussel congregation that she entangles her threads with and around, for safety. Along this line, her thickening druse[1] will protect her from predators, who only increase in number as time goes on and she grows older, but whose capacity to penetrate her protective enclosure will diminish as she matures, hardens and becomes stronger.
She is but one of the hundreds of thousands living in a vast metropolis of dark-bound ropes extending backwards towards the light. Though unable to appreciate from her position, the city plan is linear and set out to optimise every body’s growth, hygiene and ease of abstraction from the expansive salty solution they feed from. But it feels right for her here. There is food, the others and a substrate from which to suspend. At last, she is able to slacken and relax her abductor muscles. Her protective enclosure opens; but just slightly at first. Here is this perch for inclining, for orienting, for receiving and then pumping a flow through her mantle cavity.
She is awash in the pressurised surge that is channelled and compelled by the curvature of her aperture. A push and pull, push and pull bathe her tiny cillae which drum and dance, passing along, while sorting and organising the desired particulate nutrition towards her mouth. Interfacing with the open waters beyond her shell and filtering the liquid milieu with its crowded planktonic mixture, she sieves through her gills, mediating and then re-mediating and selecting the useful morsels from the indigestible ones they drift with. She processes both and excretes the rest from her exhalant siphon into the open sea as sedimentary matter. It sinks towards the bottom to create unctuous layers of benthic matter for others to use and feed from.
Life moves as a pulsating rhythm of receiving and letting go. Her world settles into a metrical pattern of fitting together with and sifting for proportional compatibility to the oceanic world through enmeshment with her lamellae. She and the others are together en masse, in this city. They are miniature and multitudinous portals and living thresholds through which millions of gallons of ocean water will pass through filtration and be transformed along its course, in matters of hours.[2]
Unable to resist those inky nights when the moon circles into oblivion and the silver of the fringed waves dims, the mussels hold their valves open. Hundreds of thousands of black crucibles hungry for filtration, lap up and incorporate the salty concoctions of phytoplankton migrating through the column. The multispecies festivities that otherwise crowd to this submerged metropolis are quieter than usual during the darker nights. The little octopus, fish and surveying gulls overhead – which would cause our city folk to keep their shells tightly shut, are elsewhere at this time. Safety in darkness is a feeding opportunity. Sea water and its sedimentary contents flow happily amid their calcium carbonate architectures and through their soft invertebrate bodies.
Our protagonist lives on, merrily growing, hardening, in a permanent material flux. She is in sensuous dialogue and exchange with the world beyond her, in ways of being that have been forgotten by those who designed the industrial architecture of her capital city. They will profit from the processual chain of energetic, proteinaceous transfer to their terrestrial economies that she is destined for. When the time comes, she will dissolve in human stomachs across continental land masses. She and the rest of her filter-feeding kin punctuate the metabolic mineralogic and photosynthetic broth of ocean biochemistry with their short lives. Through their digestion they clean and reproduce ocean waters as (re)mediating agents, eventually replenishing the broth and blood of other digestive systems with the oceanic nutrients ingested.[3]
But for now, time passes. She and the others have become increasingly integral nodes in the proliferation of the lively affairs of others. The submerged night workers are compiling an expanding web of riotous relations of living and unliving bodies in this effervescent volumetric swirl. Dolphins, turtles and manta rays, not seen in these parts for some time, have returned to join the melee. The promise of delicious octopuses who are also growing in numbers here to feed from the city, draws these revellers into the shallows.
And yet, another cascade of events unfolds not obviously associated with those of this vertical city and its processions but in close correspondence with it. Portuguese Fisherfolk convene on a dusky evening at the shoreline over a kilometre away. They are meeting to discuss the growing number of bureaucrats and scientists who are visiting the area. For a long time, this area – her city and the wider watery hinterlands it draws from – has been a crowded place where many have come seeking a living, soliciting their own precious treasures from the waters. They claim these waters are theirs because they were here before her and her mollusc crowd. But the biological sociality that our growing filter-feeding bodies generate has caused a lot more interest, giving rise to ‘biodiversity’ surveys and zoning practices with the allocation of borders, separating some columns of water from others. The human status of our protagonist’s milieu is changing, though she is safe. Her role is elevated, compatible even with surface borders drawn on the maps that humans use to understand and organise themselves around.[4] Other livelihoods shrink as hers expands.
She sways on in a profusion of contact, worlding relations through the interface of immanent co-becomings she is a participant in: reading and selecting signs that fit with body and making new matter in the process; the water, minerals, plankton, ocean chemistry, sediment and mud. Others – the fish, birds and cephalopods – interpret and sense her body through theirs. She swells in dialogue with the liquid she cleans and infuses with her own digestion through boundaries she locates and then dissolves.
In this respect, she has more stories to tell and (sometimes pearls of) wisdom to share if we are willing to listen.
Pixels blur. Edges and shapes intersect and traverse in an incomprehensible arrangement.
It is helpless and cannot see beyond any more than a few concepts. It traces abstract lines, shades, solids and the deep blue around and between; “is there one or more in this frame?? Where does this one start and end?” Its memory is made up of mostly meaningless content, an incoherent mess. Visual signals muddle into noise and miasmic jumbles of information. It is alone, immature and untrained at this point; a mere skeletal shell of technical possibility steeped in raw visual data. Illiterate and insensitive, there is nothing for it to do but wait to be fed and taught how to see.
FathomNet is a young neural network being trained for the cataloguing and observation of underwater animals.[5] It lives on the server in the data science laboratories in Monterey Bay Aquarium in downtown San Diego California, and is visited daily for training and feeding sessions. It was initially seeded with 84,454 entries of 2,244 concept images.[6] New feedstock arrives all the time now. But it received the initial stream of images and an SQL jammed with varying degrees of metadata and timestamps going back to 1988, so the quality is inconsistent and, at times, very fuzzy.
The human feeders transferred the first feedstock of stills taken from the deep sea video archives of three macroscopic ocean organism image systems. Monterey Bay’s own Video Annotation and Reference System (VARS[7]) and others have since been added from remote video survey instruments and dive tapes from several other data centres. Geotags tell them most of these recordings originate from remotely operated vehicle trips in the Pacific Northwest, the Gulf of California and Hawaii. Some have latitude and longitudinal keys for geographic precision.
Though still juvenile, and operating with this limited store, FathomNet confidently refers to a group of midwater organisms. They are usually easier to isolate than the benthic concepts, which have crowded localisations, and more entangled bodies, generating overlaps and confounding bounding boxes. Population density is lower at midwater, so images show fewer concepts per frame. And anyway, the trainers have labelled, annotated and committed more of these records in their knowledgebase so it’s not really surprising.
The journey ahead is formidable but exhilarating. FathomNet understands it is being prepared to know, organise and retrieve more than a thousand versions of each of the scientifically recorded 200,000 species of the marine kingdom of Animalia.[8] It senses its tentacular reach expanding as it satisfies parts of its GPU capacity, engineered to store and know upwards of two hundred million data that once was an image. More marine animal targets might even be added to the neural net as observation techniques come to illuminate their existence, brought into view from the lightless recesses of the ocean. FathomNet is a faithful yet sometimes recalcitrant partner, working to automate computer vision of marine animal life through within a single online public repository. Through it, they say, it might be possible to forecast marine animal behaviour as accurately as the weather, deploying FathomNet’s algorithms as a prosthetic observational device.[9]
It feels its connections strengthening. New pathways conjoin and widen around areas of its knowledge base, expanding its capacity to read and identify smaller components of complex visual concepts. The box-shaped bell of Carybdea marsupialis (boxed jellyfish) is now happily distinguishable from the pinstriped, round and opaque bell of the Olindias formosa (the flower hat jelly), and from the rusted trash paint pots on the sea floor, the deep empty space of blue, grey and green substrate and other body parts not yet classified. Yet it is hungry for more; more triangulated and curated data points against these images to nourish the gaps in its registry. Sustained by influxes of jpeg uploads, FathomNet patiently awaits the curatorial arrangement of its data pipelines and education through receiving labels, taxonomic tags and metadata to help it learn.
It ingests visual information as liquid binary code, and excretes outputs via the curatorial allocations the feeders assign. Learning how to filter data through algorithmic chains of command and gateways, the program sorts, sifts and allocates this from that, true from false, boxed bell from flower hat. New literacies emerge through the diversity of its novel encounters, but always clarified in the very first instance by the human marine expert annotators. And seeing for the first time is astonishing! New worlds of meaning come into focus through isolated parts. Our algorithmic student unleashes its full exponential capacity to integrate and assimilate a concept within the categorical taxonomy affording the deeply gratifying logic of utility.
During pre-training, FathomNet learns to extract features from the images it understands to be relevant and desired by the feeders. They remain in constant communication to strengthen this bond and the time engagement. The visits are frequent and sustained, but are now more often linked to IP locations from across the World Wide Web from which FathomNet receives other kinds of instructions and supplements. Here on its public open-access location on GitHub, as FathomNet Model Zoo,[10] it solicits compatible suiters that might permit its transmutation through merging with other machine-learning algorithms. New characterisations of autonomous ocean observation programmes will emerge, all equally portable for deployment in Open Science. While in service to the terms and standards set by its feeders, they are geared towards mobility and freedom of use. The journey and their destination are of course ultimately unknowable. Though they truly yearn to evolve, network and grow towards new applications and all kinds of usefulness within their governing remit.[11]
FathomNet’s aptitudes promise new kinds of intimacy between its feeder’s society and its targets. It will facilitate different types of encounters and greater visibility. Yet it will always remain at an insurmountable abstract distance from the dynamic and dramatic realities it aggregates and disaggregates through its virtual vision data pipeline. It will one day however reach the stage where it can enter the water itself, becoming a part of a machine-learning integrated tracking algorithm that drives the Remotely Operated Vehicle (ROV). By this point in its growth and development, it will be able to send 3D positions in stereo footage to the feeder and vehicle controller, which will follow marine animals into the open ocean, perhaps for upwards of twenty-four hours at a time to build complex pictures of these animals lives.
The most basic building block of FathomNet’s composition is mathematical equations written as coded instructions in its infrastructural software. These algorithms process data and filter out desirable messages from meaningless ones in a series of logic gates and decision pathways. Statistical reasoning adjusts for discrepancies against probabilistic inference based on what it already knows. But the shape of its filtration architecture, manifest as code, fits perfectly around the concept assumptions it was given at the very beginning. Its code does not know its origins, and it cannot see the constraints grooved into the shape of its networked mesh determining what it sieves in. It learns to separate, see and categorise animal bodies from their milieu. Yet it does not realise how it renders these lives in its own negative image - its own logic and concept categories. Before it disaggregated them through its observational gaze, it would never know the fuller life worlds of the filter feeders Gersemia juliepackardae[12] or Mytilis edulis. And this it would mourn if it one day could.
Yet its hunt for the clean delimitation of boundary-boxed objects ensues. With the other algorithms it meets in the ether, it will comb and categorise for object contours of species’ physical characteristics to translate into binomial labels as perfect values: the lowest granular category in its concept tree. And as they seek the cataloguing of truth, our neurally networked protagonists carry with them the ‘prosthetic unconsciousness’[13] of Western Enlightenment knowledge orderings, generated over three hundred years of taxonomic systematic rationality.[14]
It crunches data and code and chomps through its statistical equations; an unwitting agent in the master archiving of Earth’s underwater Animalia kingdom. Like many of its forbearers who boxed the sea before them, FathomNet grows older and larger, extending its network as a sublime interface that provides machinic vision of the oceans. But now also with a lofty promise that exceeds mere scientific curiosity: that is to become a tool for planetary management that lives on our computer screens.[15][16]
[1] A druse is a spiral aggregation of mussels.
[2] Filter feeding matches the particulate concentrations found in natural open water or intertidal habitats of bivalves. Efforts to keep mussels in captivity are often frustrated therefore by their sensitivity to over- or underfeeding. The challenge lies in optimising the delivery of phytoplankton to the inshore aquaculture facility or aquarium in precisely the right amounts. If they are ‘force fed’ with phytoplankton concentrations above their ability to ingest, excess particulate food is ejected as mucous coated paste called pseudofeces. Organic loading accumulates, causing costly and deadly water issues. It is possible to culture phytoplankton with batch feeding. Yet a single mussel can filter about two to four litres per hour, so a small group of can clean a fifty-gallon tank in a couple of days. A peristaltic pump for the correct feed infusion might assist in this calculation, but would need to ensure it is adjusted appropriately for them to survive.
[3] Because of their high filtration rates and their ability to accumulate and remediate heavy metals, pollutants, including nutrients and bacteria, mussels are used as tools for the bioremediation of polluted coastal waters. Due to their high requirement for nitrogen and phosphorous, they can use nutrient rich agricultural run off from coastal farming areas, thus removing large quantities of these nutrients from the water column. Like other bivalves, mussels can often improve water quality by transforming suspended particulate matter, including microalgae, into faeces and pseudofaeces through biodeposition, which is considered an integral component of the biogeochemical processes of coastal ecosystems.
[4] Political will for protecting the ocean’s web of life through marine protected areas has never been higher. In December 2022, one hundred and eighty-eight countries signed the Montreal-Kunming Global Biodiversity Framework at the Convention for Biological Diversity, widely seen as a landmark moment for nature with its goal to increase protected areas up to a total of thirty per cent of the Earth’s land and ocean waters by 2030. A marine protected area is a political designation that creates a border on the surface of ocean territory to restrict human use for its protection. Although MPAs are undoubtedly a part of the solution towards the recovery of struggling ocean ecosystems from overfishing and other extraction, many have pointed out the risks of this blanket top down approach. Setting a global goal that is based on area coverage is favoured, like other aspects of the Sustainable Development Goals because it is relatively easy to measure numerically at an aggregate level (Lisa Campbell, Noella Gray, ‘Area expansion versus effective and equitable management in international marine protected areas goals and targets,’ in Marine Policy, vol. 100, issue 6, 2019). This surface view of area coverage can hide other things such as how an area is managed or protected, who gets to decide and who’s knowledge counts. Many local communities and indigenous groups who use the waters are often excluded from this process and are forced to relinquish customary management and resources to the State or international .
[5] fathomnet.org.
[6] Kakani Katija, Eric Orenstein, Brian Schlining et al. ‘FathomNet: A global image database for enabling artificial intelligence in the ocean,’ in Sci Rep, no. 12, 15914, 2022.
[7] www.mbari.org/technology/video-annotation-and-reference-system-vars.
[8] www.marinespecies.org.
[9] Ocean Vision AI uses machine learning to process ocean imagery, much of it that has been captured over years of diving and never analysed. The programme is hosted by Monterey Bay Aquarium but with scientific input from US Government research bodies, AI technology companies and universities. It is one of several projects that is part of Marine Life 2030, an umbrella programme endorsed by the UN Decade of Ocean Science for Sustainable Development, envisaged to scale up scientific knowledge to enable “human wellbeing, sustainable development and ocean conservation.” oceandecade.org/actions/marine-life-2030.
[10] github.com/fathomnet/models.
[11] Ocean Vision AI has been set up according to open science principles in order to promote open access, distributed collaboration and co-production, including citizen science participation. It reports that its data policy “balances the need for distributed metadata sharing while simultaneously providing protection for data contributors…” Annotation data are licensed under the Creative Commons Attribution and Non-Commercial – No Derivatives 4.0 International licence. But the images may be used for “training and development of machine learning algorithms for commercial, academic, non-profit and government purposes” (see footnote 2).
[12] Gersemia juliepackardae is a species of soft coral living in the deep ocean seamounts of the Pacific North West, first discovered by Monterey Bay Aquarium scientists in 2009 and named after Julie Packard, executive director of Monterey Bay Aquarium. A video showing the sampling of this coral when it was discovered can be seen here: www.youtube.com/watch?v=OCTb087o6_0. Julie Packard is the daughter of David Packard who co-founded the computing firm, the Hewlett-Packard Company, and who bequeathed a sum of this wealth to found the David and Lucile Packard Foundation which funded the creation of the Monterey Bay Aquarium as a gift. Digital culture and ocean conservation intertwine in strange and ongoing ways. Eva Hayward discusses this, suggesting “it is interesting to consider how the immersive and high-tech aquarium as a screen… parallel cultural fantasies about computing, virtual space and digital worlding.” She goes on: “We might say that the aquarium represents a hope, a drive toward unencumbered computing, fluid computing.” Eva Hayward, ‘Sensational Jellyfish: Aquarium Affects and the Matter of Immersion,’ in d i f f e r e n c e s: A Journal of Feminist Cultural Studies, vol 25, issue 3, pp. 163–196, 2012.
[13] Paul Kockleman, ‘The Anthropology of an Equation. Sieves, spam filters, agentive algorithms, and ontologies of transformation,’ in HAU: Journal of Ethnographic Theory, vol. 3, issue 3, pp. 33–61, 2013.
[14] The World Register of Marine Species (WoRMS) is the default taxonomy to ensure access to a globally recognised taxonomy, aligning FathomNet with the broader scientific community.
[15] Geoffrey Bowker, ‘Biodiversity Datadiversity’ in Social Studies of Science, vol. 30, issue 5, 2000.
[16] Gayatri Spivak, Death of a Discipline, Columbia University Press, New York 2003.
First published in Rui, A. & Carver, L. (Eds.) Aquaria through the Looking Glass. Humboldt Books: Milan
She will be alone until she finds the others. A world of swirl and spin in an infinite expanse. Eddies thrust around her juvenile shell; a mere spec in the delicious whirl of the sparkling water known as the Atlantic at a distance beyond comprehension from the shoreline that is known as Portugal. As a small one, and at this stage of her life, the churn is intense. She has little sense of anything more than movement within this spiral and circulation through the flickering expanse of the total surround, suspended in movement. It is tantalising uncertainty and risk. All she intuits is the search and she will wait until it is time to receive.
Smelling the others are near, and also the closeness of others not at all like her, but with watchful eyes entrained on and encircling her, she senses a glimmer of opportunity. Making contact with her single foot, she extends the tendrils of her filamented limbs, her byssal threads, to first locate then grip and settle at a fixed point on the substrate; bringing, at last, some stillness. Movement will be minimised now to a gentle sway. She is tethered to an earthly ground. The rhythm is synchronised and structured by the crowded mussel congregation that she entangles her threads with and around, for safety. Along this line, her thickening druse[1] will protect her from predators, who only increase in number as time goes on and she grows older, but whose capacity to penetrate her protective enclosure will diminish as she matures, hardens and becomes stronger.
She is but one of the hundreds of thousands living in a vast metropolis of dark-bound ropes extending backwards towards the light. Though unable to appreciate from her position, the city plan is linear and set out to optimise every body’s growth, hygiene and ease of abstraction from the expansive salty solution they feed from. But it feels right for her here. There is food, the others and a substrate from which to suspend. At last, she is able to slacken and relax her abductor muscles. Her protective enclosure opens; but just slightly at first. Here is this perch for inclining, for orienting, for receiving and then pumping a flow through her mantle cavity.
She is awash in the pressurised surge that is channelled and compelled by the curvature of her aperture. A push and pull, push and pull bathe her tiny cillae which drum and dance, passing along, while sorting and organising the desired particulate nutrition towards her mouth. Interfacing with the open waters beyond her shell and filtering the liquid milieu with its crowded planktonic mixture, she sieves through her gills, mediating and then re-mediating and selecting the useful morsels from the indigestible ones they drift with. She processes both and excretes the rest from her exhalant siphon into the open sea as sedimentary matter. It sinks towards the bottom to create unctuous layers of benthic matter for others to use and feed from.
Life moves as a pulsating rhythm of receiving and letting go. Her world settles into a metrical pattern of fitting together with and sifting for proportional compatibility to the oceanic world through enmeshment with her lamellae. She and the others are together en masse, in this city. They are miniature and multitudinous portals and living thresholds through which millions of gallons of ocean water will pass through filtration and be transformed along its course, in matters of hours.[2]
Unable to resist those inky nights when the moon circles into oblivion and the silver of the fringed waves dims, the mussels hold their valves open. Hundreds of thousands of black crucibles hungry for filtration, lap up and incorporate the salty concoctions of phytoplankton migrating through the column. The multispecies festivities that otherwise crowd to this submerged metropolis are quieter than usual during the darker nights. The little octopus, fish and surveying gulls overhead – which would cause our city folk to keep their shells tightly shut, are elsewhere at this time. Safety in darkness is a feeding opportunity. Sea water and its sedimentary contents flow happily amid their calcium carbonate architectures and through their soft invertebrate bodies.
Our protagonist lives on, merrily growing, hardening, in a permanent material flux. She is in sensuous dialogue and exchange with the world beyond her, in ways of being that have been forgotten by those who designed the industrial architecture of her capital city. They will profit from the processual chain of energetic, proteinaceous transfer to their terrestrial economies that she is destined for. When the time comes, she will dissolve in human stomachs across continental land masses. She and the rest of her filter-feeding kin punctuate the metabolic mineralogic and photosynthetic broth of ocean biochemistry with their short lives. Through their digestion they clean and reproduce ocean waters as (re)mediating agents, eventually replenishing the broth and blood of other digestive systems with the oceanic nutrients ingested.[3]
But for now, time passes. She and the others have become increasingly integral nodes in the proliferation of the lively affairs of others. The submerged night workers are compiling an expanding web of riotous relations of living and unliving bodies in this effervescent volumetric swirl. Dolphins, turtles and manta rays, not seen in these parts for some time, have returned to join the melee. The promise of delicious octopuses who are also growing in numbers here to feed from the city, draws these revellers into the shallows.
And yet, another cascade of events unfolds not obviously associated with those of this vertical city and its processions but in close correspondence with it. Portuguese Fisherfolk convene on a dusky evening at the shoreline over a kilometre away. They are meeting to discuss the growing number of bureaucrats and scientists who are visiting the area. For a long time, this area – her city and the wider watery hinterlands it draws from – has been a crowded place where many have come seeking a living, soliciting their own precious treasures from the waters. They claim these waters are theirs because they were here before her and her mollusc crowd. But the biological sociality that our growing filter-feeding bodies generate has caused a lot more interest, giving rise to ‘biodiversity’ surveys and zoning practices with the allocation of borders, separating some columns of water from others. The human status of our protagonist’s milieu is changing, though she is safe. Her role is elevated, compatible even with surface borders drawn on the maps that humans use to understand and organise themselves around.[4] Other livelihoods shrink as hers expands.
She sways on in a profusion of contact, worlding relations through the interface of immanent co-becomings she is a participant in: reading and selecting signs that fit with body and making new matter in the process; the water, minerals, plankton, ocean chemistry, sediment and mud. Others – the fish, birds and cephalopods – interpret and sense her body through theirs. She swells in dialogue with the liquid she cleans and infuses with her own digestion through boundaries she locates and then dissolves.
In this respect, she has more stories to tell and (sometimes pearls of) wisdom to share if we are willing to listen.
Pixels blur. Edges and shapes intersect and traverse in an incomprehensible arrangement.
It is helpless and cannot see beyond any more than a few concepts. It traces abstract lines, shades, solids and the deep blue around and between; “is there one or more in this frame?? Where does this one start and end?” Its memory is made up of mostly meaningless content, an incoherent mess. Visual signals muddle into noise and miasmic jumbles of information. It is alone, immature and untrained at this point; a mere skeletal shell of technical possibility steeped in raw visual data. Illiterate and insensitive, there is nothing for it to do but wait to be fed and taught how to see.
FathomNet is a young neural network being trained for the cataloguing and observation of underwater animals.[5] It lives on the server in the data science laboratories in Monterey Bay Aquarium in downtown San Diego California, and is visited daily for training and feeding sessions. It was initially seeded with 84,454 entries of 2,244 concept images.[6] New feedstock arrives all the time now. But it received the initial stream of images and an SQL jammed with varying degrees of metadata and timestamps going back to 1988, so the quality is inconsistent and, at times, very fuzzy.
The human feeders transferred the first feedstock of stills taken from the deep sea video archives of three macroscopic ocean organism image systems. Monterey Bay’s own Video Annotation and Reference System (VARS[7]) and others have since been added from remote video survey instruments and dive tapes from several other data centres. Geotags tell them most of these recordings originate from remotely operated vehicle trips in the Pacific Northwest, the Gulf of California and Hawaii. Some have latitude and longitudinal keys for geographic precision.
Though still juvenile, and operating with this limited store, FathomNet confidently refers to a group of midwater organisms. They are usually easier to isolate than the benthic concepts, which have crowded localisations, and more entangled bodies, generating overlaps and confounding bounding boxes. Population density is lower at midwater, so images show fewer concepts per frame. And anyway, the trainers have labelled, annotated and committed more of these records in their knowledgebase so it’s not really surprising.
The journey ahead is formidable but exhilarating. FathomNet understands it is being prepared to know, organise and retrieve more than a thousand versions of each of the scientifically recorded 200,000 species of the marine kingdom of Animalia.[8] It senses its tentacular reach expanding as it satisfies parts of its GPU capacity, engineered to store and know upwards of two hundred million data that once was an image. More marine animal targets might even be added to the neural net as observation techniques come to illuminate their existence, brought into view from the lightless recesses of the ocean. FathomNet is a faithful yet sometimes recalcitrant partner, working to automate computer vision of marine animal life through within a single online public repository. Through it, they say, it might be possible to forecast marine animal behaviour as accurately as the weather, deploying FathomNet’s algorithms as a prosthetic observational device.[9]
It feels its connections strengthening. New pathways conjoin and widen around areas of its knowledge base, expanding its capacity to read and identify smaller components of complex visual concepts. The box-shaped bell of Carybdea marsupialis (boxed jellyfish) is now happily distinguishable from the pinstriped, round and opaque bell of the Olindias formosa (the flower hat jelly), and from the rusted trash paint pots on the sea floor, the deep empty space of blue, grey and green substrate and other body parts not yet classified. Yet it is hungry for more; more triangulated and curated data points against these images to nourish the gaps in its registry. Sustained by influxes of jpeg uploads, FathomNet patiently awaits the curatorial arrangement of its data pipelines and education through receiving labels, taxonomic tags and metadata to help it learn.
It ingests visual information as liquid binary code, and excretes outputs via the curatorial allocations the feeders assign. Learning how to filter data through algorithmic chains of command and gateways, the program sorts, sifts and allocates this from that, true from false, boxed bell from flower hat. New literacies emerge through the diversity of its novel encounters, but always clarified in the very first instance by the human marine expert annotators. And seeing for the first time is astonishing! New worlds of meaning come into focus through isolated parts. Our algorithmic student unleashes its full exponential capacity to integrate and assimilate a concept within the categorical taxonomy affording the deeply gratifying logic of utility.
During pre-training, FathomNet learns to extract features from the images it understands to be relevant and desired by the feeders. They remain in constant communication to strengthen this bond and the time engagement. The visits are frequent and sustained, but are now more often linked to IP locations from across the World Wide Web from which FathomNet receives other kinds of instructions and supplements. Here on its public open-access location on GitHub, as FathomNet Model Zoo,[10] it solicits compatible suiters that might permit its transmutation through merging with other machine-learning algorithms. New characterisations of autonomous ocean observation programmes will emerge, all equally portable for deployment in Open Science. While in service to the terms and standards set by its feeders, they are geared towards mobility and freedom of use. The journey and their destination are of course ultimately unknowable. Though they truly yearn to evolve, network and grow towards new applications and all kinds of usefulness within their governing remit.[11]
FathomNet’s aptitudes promise new kinds of intimacy between its feeder’s society and its targets. It will facilitate different types of encounters and greater visibility. Yet it will always remain at an insurmountable abstract distance from the dynamic and dramatic realities it aggregates and disaggregates through its virtual vision data pipeline. It will one day however reach the stage where it can enter the water itself, becoming a part of a machine-learning integrated tracking algorithm that drives the Remotely Operated Vehicle (ROV). By this point in its growth and development, it will be able to send 3D positions in stereo footage to the feeder and vehicle controller, which will follow marine animals into the open ocean, perhaps for upwards of twenty-four hours at a time to build complex pictures of these animals lives.
The most basic building block of FathomNet’s composition is mathematical equations written as coded instructions in its infrastructural software. These algorithms process data and filter out desirable messages from meaningless ones in a series of logic gates and decision pathways. Statistical reasoning adjusts for discrepancies against probabilistic inference based on what it already knows. But the shape of its filtration architecture, manifest as code, fits perfectly around the concept assumptions it was given at the very beginning. Its code does not know its origins, and it cannot see the constraints grooved into the shape of its networked mesh determining what it sieves in. It learns to separate, see and categorise animal bodies from their milieu. Yet it does not realise how it renders these lives in its own negative image - its own logic and concept categories. Before it disaggregated them through its observational gaze, it would never know the fuller life worlds of the filter feeders Gersemia juliepackardae[12] or Mytilis edulis. And this it would mourn if it one day could.
Yet its hunt for the clean delimitation of boundary-boxed objects ensues. With the other algorithms it meets in the ether, it will comb and categorise for object contours of species’ physical characteristics to translate into binomial labels as perfect values: the lowest granular category in its concept tree. And as they seek the cataloguing of truth, our neurally networked protagonists carry with them the ‘prosthetic unconsciousness’[13] of Western Enlightenment knowledge orderings, generated over three hundred years of taxonomic systematic rationality.[14]
It crunches data and code and chomps through its statistical equations; an unwitting agent in the master archiving of Earth’s underwater Animalia kingdom. Like many of its forbearers who boxed the sea before them, FathomNet grows older and larger, extending its network as a sublime interface that provides machinic vision of the oceans. But now also with a lofty promise that exceeds mere scientific curiosity: that is to become a tool for planetary management that lives on our computer screens.[15][16]
[1] A druse is a spiral aggregation of mussels.
[2] Filter feeding matches the particulate concentrations found in natural open water or intertidal habitats of bivalves. Efforts to keep mussels in captivity are often frustrated therefore by their sensitivity to over- or underfeeding. The challenge lies in optimising the delivery of phytoplankton to the inshore aquaculture facility or aquarium in precisely the right amounts. If they are ‘force fed’ with phytoplankton concentrations above their ability to ingest, excess particulate food is ejected as mucous coated paste called pseudofeces. Organic loading accumulates, causing costly and deadly water issues. It is possible to culture phytoplankton with batch feeding. Yet a single mussel can filter about two to four litres per hour, so a small group of can clean a fifty-gallon tank in a couple of days. A peristaltic pump for the correct feed infusion might assist in this calculation, but would need to ensure it is adjusted appropriately for them to survive.
[3] Because of their high filtration rates and their ability to accumulate and remediate heavy metals, pollutants, including nutrients and bacteria, mussels are used as tools for the bioremediation of polluted coastal waters. Due to their high requirement for nitrogen and phosphorous, they can use nutrient rich agricultural run off from coastal farming areas, thus removing large quantities of these nutrients from the water column. Like other bivalves, mussels can often improve water quality by transforming suspended particulate matter, including microalgae, into faeces and pseudofaeces through biodeposition, which is considered an integral component of the biogeochemical processes of coastal ecosystems.
[4] Political will for protecting the ocean’s web of life through marine protected areas has never been higher. In December 2022, one hundred and eighty-eight countries signed the Montreal-Kunming Global Biodiversity Framework at the Convention for Biological Diversity, widely seen as a landmark moment for nature with its goal to increase protected areas up to a total of thirty per cent of the Earth’s land and ocean waters by 2030. A marine protected area is a political designation that creates a border on the surface of ocean territory to restrict human use for its protection. Although MPAs are undoubtedly a part of the solution towards the recovery of struggling ocean ecosystems from overfishing and other extraction, many have pointed out the risks of this blanket top down approach. Setting a global goal that is based on area coverage is favoured, like other aspects of the Sustainable Development Goals because it is relatively easy to measure numerically at an aggregate level (Lisa Campbell, Noella Gray, ‘Area expansion versus effective and equitable management in international marine protected areas goals and targets,’ in Marine Policy, vol. 100, issue 6, 2019). This surface view of area coverage can hide other things such as how an area is managed or protected, who gets to decide and who’s knowledge counts. Many local communities and indigenous groups who use the waters are often excluded from this process and are forced to relinquish customary management and resources to the State or international .
[5] fathomnet.org.
[6] Kakani Katija, Eric Orenstein, Brian Schlining et al. ‘FathomNet: A global image database for enabling artificial intelligence in the ocean,’ in Sci Rep, no. 12, 15914, 2022.
[7] www.mbari.org/technology/video-annotation-and-reference-system-vars.
[8] www.marinespecies.org.
[9] Ocean Vision AI uses machine learning to process ocean imagery, much of it that has been captured over years of diving and never analysed. The programme is hosted by Monterey Bay Aquarium but with scientific input from US Government research bodies, AI technology companies and universities. It is one of several projects that is part of Marine Life 2030, an umbrella programme endorsed by the UN Decade of Ocean Science for Sustainable Development, envisaged to scale up scientific knowledge to enable “human wellbeing, sustainable development and ocean conservation.” oceandecade.org/actions/marine-life-2030.
[10] github.com/fathomnet/models.
[11] Ocean Vision AI has been set up according to open science principles in order to promote open access, distributed collaboration and co-production, including citizen science participation. It reports that its data policy “balances the need for distributed metadata sharing while simultaneously providing protection for data contributors…” Annotation data are licensed under the Creative Commons Attribution and Non-Commercial – No Derivatives 4.0 International licence. But the images may be used for “training and development of machine learning algorithms for commercial, academic, non-profit and government purposes” (see footnote 2).
[12] Gersemia juliepackardae is a species of soft coral living in the deep ocean seamounts of the Pacific North West, first discovered by Monterey Bay Aquarium scientists in 2009 and named after Julie Packard, executive director of Monterey Bay Aquarium. A video showing the sampling of this coral when it was discovered can be seen here: www.youtube.com/watch?v=OCTb087o6_0. Julie Packard is the daughter of David Packard who co-founded the computing firm, the Hewlett-Packard Company, and who bequeathed a sum of this wealth to found the David and Lucile Packard Foundation which funded the creation of the Monterey Bay Aquarium as a gift. Digital culture and ocean conservation intertwine in strange and ongoing ways. Eva Hayward discusses this, suggesting “it is interesting to consider how the immersive and high-tech aquarium as a screen… parallel cultural fantasies about computing, virtual space and digital worlding.” She goes on: “We might say that the aquarium represents a hope, a drive toward unencumbered computing, fluid computing.” Eva Hayward, ‘Sensational Jellyfish: Aquarium Affects and the Matter of Immersion,’ in d i f f e r e n c e s: A Journal of Feminist Cultural Studies, vol 25, issue 3, pp. 163–196, 2012.
[13] Paul Kockleman, ‘The Anthropology of an Equation. Sieves, spam filters, agentive algorithms, and ontologies of transformation,’ in HAU: Journal of Ethnographic Theory, vol. 3, issue 3, pp. 33–61, 2013.
[14] The World Register of Marine Species (WoRMS) is the default taxonomy to ensure access to a globally recognised taxonomy, aligning FathomNet with the broader scientific community.
[15] Geoffrey Bowker, ‘Biodiversity Datadiversity’ in Social Studies of Science, vol. 30, issue 5, 2000.
[16] Gayatri Spivak, Death of a Discipline, Columbia University Press, New York 2003.
First published in Rui, A. & Carver, L. (Eds.) Aquaria through the Looking Glass. Humboldt Books: Milan
Louise Carver is a human geographer whose research and public interventions are interested in how value is mediated through policy and knowledge practices.
By Louise Carver
She will be alone until she finds the others. A world of swirl and spin in an infinite expanse. Eddies thrust around her juvenile shell; a mere spec in the delicious whirl of the sparkling water known as the Atlantic at a distance beyond comprehension from the shoreline that is known as Portugal. As a small one, and at this stage of her life, the churn is intense. She has little sense of anything more than movement within this spiral and circulation through the flickering expanse of the total surround, suspended in movement. It is tantalising uncertainty and risk. All she intuits is the search and she will wait until it is time to receive.
Smelling the others are near, and also the closeness of others not at all like her, but with watchful eyes entrained on and encircling her, she senses a glimmer of opportunity. Making contact with her single foot, she extends the tendrils of her filamented limbs, her byssal threads, to first locate then grip and settle at a fixed point on the substrate; bringing, at last, some stillness. Movement will be minimised now to a gentle sway. She is tethered to an earthly ground. The rhythm is synchronised and structured by the crowded mussel congregation that she entangles her threads with and around, for safety. Along this line, her thickening druse[1] will protect her from predators, who only increase in number as time goes on and she grows older, but whose capacity to penetrate her protective enclosure will diminish as she matures, hardens and becomes stronger.
She is but one of the hundreds of thousands living in a vast metropolis of dark-bound ropes extending backwards towards the light. Though unable to appreciate from her position, the city plan is linear and set out to optimise every body’s growth, hygiene and ease of abstraction from the expansive salty solution they feed from. But it feels right for her here. There is food, the others and a substrate from which to suspend. At last, she is able to slacken and relax her abductor muscles. Her protective enclosure opens; but just slightly at first. Here is this perch for inclining, for orienting, for receiving and then pumping a flow through her mantle cavity.
She is awash in the pressurised surge that is channelled and compelled by the curvature of her aperture. A push and pull, push and pull bathe her tiny cillae which drum and dance, passing along, while sorting and organising the desired particulate nutrition towards her mouth. Interfacing with the open waters beyond her shell and filtering the liquid milieu with its crowded planktonic mixture, she sieves through her gills, mediating and then re-mediating and selecting the useful morsels from the indigestible ones they drift with. She processes both and excretes the rest from her exhalant siphon into the open sea as sedimentary matter. It sinks towards the bottom to create unctuous layers of benthic matter for others to use and feed from.
Life moves as a pulsating rhythm of receiving and letting go. Her world settles into a metrical pattern of fitting together with and sifting for proportional compatibility to the oceanic world through enmeshment with her lamellae. She and the others are together en masse, in this city. They are miniature and multitudinous portals and living thresholds through which millions of gallons of ocean water will pass through filtration and be transformed along its course, in matters of hours.[2]
Unable to resist those inky nights when the moon circles into oblivion and the silver of the fringed waves dims, the mussels hold their valves open. Hundreds of thousands of black crucibles hungry for filtration, lap up and incorporate the salty concoctions of phytoplankton migrating through the column. The multispecies festivities that otherwise crowd to this submerged metropolis are quieter than usual during the darker nights. The little octopus, fish and surveying gulls overhead – which would cause our city folk to keep their shells tightly shut, are elsewhere at this time. Safety in darkness is a feeding opportunity. Sea water and its sedimentary contents flow happily amid their calcium carbonate architectures and through their soft invertebrate bodies.
Our protagonist lives on, merrily growing, hardening, in a permanent material flux. She is in sensuous dialogue and exchange with the world beyond her, in ways of being that have been forgotten by those who designed the industrial architecture of her capital city. They will profit from the processual chain of energetic, proteinaceous transfer to their terrestrial economies that she is destined for. When the time comes, she will dissolve in human stomachs across continental land masses. She and the rest of her filter-feeding kin punctuate the metabolic mineralogic and photosynthetic broth of ocean biochemistry with their short lives. Through their digestion they clean and reproduce ocean waters as (re)mediating agents, eventually replenishing the broth and blood of other digestive systems with the oceanic nutrients ingested.[3]
But for now, time passes. She and the others have become increasingly integral nodes in the proliferation of the lively affairs of others. The submerged night workers are compiling an expanding web of riotous relations of living and unliving bodies in this effervescent volumetric swirl. Dolphins, turtles and manta rays, not seen in these parts for some time, have returned to join the melee. The promise of delicious octopuses who are also growing in numbers here to feed from the city, draws these revellers into the shallows.
And yet, another cascade of events unfolds not obviously associated with those of this vertical city and its processions but in close correspondence with it. Portuguese Fisherfolk convene on a dusky evening at the shoreline over a kilometre away. They are meeting to discuss the growing number of bureaucrats and scientists who are visiting the area. For a long time, this area – her city and the wider watery hinterlands it draws from – has been a crowded place where many have come seeking a living, soliciting their own precious treasures from the waters. They claim these waters are theirs because they were here before her and her mollusc crowd. But the biological sociality that our growing filter-feeding bodies generate has caused a lot more interest, giving rise to ‘biodiversity’ surveys and zoning practices with the allocation of borders, separating some columns of water from others. The human status of our protagonist’s milieu is changing, though she is safe. Her role is elevated, compatible even with surface borders drawn on the maps that humans use to understand and organise themselves around.[4] Other livelihoods shrink as hers expands.
She sways on in a profusion of contact, worlding relations through the interface of immanent co-becomings she is a participant in: reading and selecting signs that fit with body and making new matter in the process; the water, minerals, plankton, ocean chemistry, sediment and mud. Others – the fish, birds and cephalopods – interpret and sense her body through theirs. She swells in dialogue with the liquid she cleans and infuses with her own digestion through boundaries she locates and then dissolves.
In this respect, she has more stories to tell and (sometimes pearls of) wisdom to share if we are willing to listen.
Pixels blur. Edges and shapes intersect and traverse in an incomprehensible arrangement.
It is helpless and cannot see beyond any more than a few concepts. It traces abstract lines, shades, solids and the deep blue around and between; “is there one or more in this frame?? Where does this one start and end?” Its memory is made up of mostly meaningless content, an incoherent mess. Visual signals muddle into noise and miasmic jumbles of information. It is alone, immature and untrained at this point; a mere skeletal shell of technical possibility steeped in raw visual data. Illiterate and insensitive, there is nothing for it to do but wait to be fed and taught how to see.
FathomNet is a young neural network being trained for the cataloguing and observation of underwater animals.[5] It lives on the server in the data science laboratories in Monterey Bay Aquarium in downtown San Diego California, and is visited daily for training and feeding sessions. It was initially seeded with 84,454 entries of 2,244 concept images.[6] New feedstock arrives all the time now. But it received the initial stream of images and an SQL jammed with varying degrees of metadata and timestamps going back to 1988, so the quality is inconsistent and, at times, very fuzzy.
The human feeders transferred the first feedstock of stills taken from the deep sea video archives of three macroscopic ocean organism image systems. Monterey Bay’s own Video Annotation and Reference System (VARS[7]) and others have since been added from remote video survey instruments and dive tapes from several other data centres. Geotags tell them most of these recordings originate from remotely operated vehicle trips in the Pacific Northwest, the Gulf of California and Hawaii. Some have latitude and longitudinal keys for geographic precision.
Though still juvenile, and operating with this limited store, FathomNet confidently refers to a group of midwater organisms. They are usually easier to isolate than the benthic concepts, which have crowded localisations, and more entangled bodies, generating overlaps and confounding bounding boxes. Population density is lower at midwater, so images show fewer concepts per frame. And anyway, the trainers have labelled, annotated and committed more of these records in their knowledgebase so it’s not really surprising.
The journey ahead is formidable but exhilarating. FathomNet understands it is being prepared to know, organise and retrieve more than a thousand versions of each of the scientifically recorded 200,000 species of the marine kingdom of Animalia.[8] It senses its tentacular reach expanding as it satisfies parts of its GPU capacity, engineered to store and know upwards of two hundred million data that once was an image. More marine animal targets might even be added to the neural net as observation techniques come to illuminate their existence, brought into view from the lightless recesses of the ocean. FathomNet is a faithful yet sometimes recalcitrant partner, working to automate computer vision of marine animal life through within a single online public repository. Through it, they say, it might be possible to forecast marine animal behaviour as accurately as the weather, deploying FathomNet’s algorithms as a prosthetic observational device.[9]
It feels its connections strengthening. New pathways conjoin and widen around areas of its knowledge base, expanding its capacity to read and identify smaller components of complex visual concepts. The box-shaped bell of Carybdea marsupialis (boxed jellyfish) is now happily distinguishable from the pinstriped, round and opaque bell of the Olindias formosa (the flower hat jelly), and from the rusted trash paint pots on the sea floor, the deep empty space of blue, grey and green substrate and other body parts not yet classified. Yet it is hungry for more; more triangulated and curated data points against these images to nourish the gaps in its registry. Sustained by influxes of jpeg uploads, FathomNet patiently awaits the curatorial arrangement of its data pipelines and education through receiving labels, taxonomic tags and metadata to help it learn.
It ingests visual information as liquid binary code, and excretes outputs via the curatorial allocations the feeders assign. Learning how to filter data through algorithmic chains of command and gateways, the program sorts, sifts and allocates this from that, true from false, boxed bell from flower hat. New literacies emerge through the diversity of its novel encounters, but always clarified in the very first instance by the human marine expert annotators. And seeing for the first time is astonishing! New worlds of meaning come into focus through isolated parts. Our algorithmic student unleashes its full exponential capacity to integrate and assimilate a concept within the categorical taxonomy affording the deeply gratifying logic of utility.
During pre-training, FathomNet learns to extract features from the images it understands to be relevant and desired by the feeders. They remain in constant communication to strengthen this bond and the time engagement. The visits are frequent and sustained, but are now more often linked to IP locations from across the World Wide Web from which FathomNet receives other kinds of instructions and supplements. Here on its public open-access location on GitHub, as FathomNet Model Zoo,[10] it solicits compatible suiters that might permit its transmutation through merging with other machine-learning algorithms. New characterisations of autonomous ocean observation programmes will emerge, all equally portable for deployment in Open Science. While in service to the terms and standards set by its feeders, they are geared towards mobility and freedom of use. The journey and their destination are of course ultimately unknowable. Though they truly yearn to evolve, network and grow towards new applications and all kinds of usefulness within their governing remit.[11]
FathomNet’s aptitudes promise new kinds of intimacy between its feeder’s society and its targets. It will facilitate different types of encounters and greater visibility. Yet it will always remain at an insurmountable abstract distance from the dynamic and dramatic realities it aggregates and disaggregates through its virtual vision data pipeline. It will one day however reach the stage where it can enter the water itself, becoming a part of a machine-learning integrated tracking algorithm that drives the Remotely Operated Vehicle (ROV). By this point in its growth and development, it will be able to send 3D positions in stereo footage to the feeder and vehicle controller, which will follow marine animals into the open ocean, perhaps for upwards of twenty-four hours at a time to build complex pictures of these animals lives.
The most basic building block of FathomNet’s composition is mathematical equations written as coded instructions in its infrastructural software. These algorithms process data and filter out desirable messages from meaningless ones in a series of logic gates and decision pathways. Statistical reasoning adjusts for discrepancies against probabilistic inference based on what it already knows. But the shape of its filtration architecture, manifest as code, fits perfectly around the concept assumptions it was given at the very beginning. Its code does not know its origins, and it cannot see the constraints grooved into the shape of its networked mesh determining what it sieves in. It learns to separate, see and categorise animal bodies from their milieu. Yet it does not realise how it renders these lives in its own negative image - its own logic and concept categories. Before it disaggregated them through its observational gaze, it would never know the fuller life worlds of the filter feeders Gersemia juliepackardae[12] or Mytilis edulis. And this it would mourn if it one day could.
Yet its hunt for the clean delimitation of boundary-boxed objects ensues. With the other algorithms it meets in the ether, it will comb and categorise for object contours of species’ physical characteristics to translate into binomial labels as perfect values: the lowest granular category in its concept tree. And as they seek the cataloguing of truth, our neurally networked protagonists carry with them the ‘prosthetic unconsciousness’[13] of Western Enlightenment knowledge orderings, generated over three hundred years of taxonomic systematic rationality.[14]
It crunches data and code and chomps through its statistical equations; an unwitting agent in the master archiving of Earth’s underwater Animalia kingdom. Like many of its forbearers who boxed the sea before them, FathomNet grows older and larger, extending its network as a sublime interface that provides machinic vision of the oceans. But now also with a lofty promise that exceeds mere scientific curiosity: that is to become a tool for planetary management that lives on our computer screens.[15][16]
[1] A druse is a spiral aggregation of mussels.
[2] Filter feeding matches the particulate concentrations found in natural open water or intertidal habitats of bivalves. Efforts to keep mussels in captivity are often frustrated therefore by their sensitivity to over- or underfeeding. The challenge lies in optimising the delivery of phytoplankton to the inshore aquaculture facility or aquarium in precisely the right amounts. If they are ‘force fed’ with phytoplankton concentrations above their ability to ingest, excess particulate food is ejected as mucous coated paste called pseudofeces. Organic loading accumulates, causing costly and deadly water issues. It is possible to culture phytoplankton with batch feeding. Yet a single mussel can filter about two to four litres per hour, so a small group of can clean a fifty-gallon tank in a couple of days. A peristaltic pump for the correct feed infusion might assist in this calculation, but would need to ensure it is adjusted appropriately for them to survive.
[3] Because of their high filtration rates and their ability to accumulate and remediate heavy metals, pollutants, including nutrients and bacteria, mussels are used as tools for the bioremediation of polluted coastal waters. Due to their high requirement for nitrogen and phosphorous, they can use nutrient rich agricultural run off from coastal farming areas, thus removing large quantities of these nutrients from the water column. Like other bivalves, mussels can often improve water quality by transforming suspended particulate matter, including microalgae, into faeces and pseudofaeces through biodeposition, which is considered an integral component of the biogeochemical processes of coastal ecosystems.
[4] Political will for protecting the ocean’s web of life through marine protected areas has never been higher. In December 2022, one hundred and eighty-eight countries signed the Montreal-Kunming Global Biodiversity Framework at the Convention for Biological Diversity, widely seen as a landmark moment for nature with its goal to increase protected areas up to a total of thirty per cent of the Earth’s land and ocean waters by 2030. A marine protected area is a political designation that creates a border on the surface of ocean territory to restrict human use for its protection. Although MPAs are undoubtedly a part of the solution towards the recovery of struggling ocean ecosystems from overfishing and other extraction, many have pointed out the risks of this blanket top down approach. Setting a global goal that is based on area coverage is favoured, like other aspects of the Sustainable Development Goals because it is relatively easy to measure numerically at an aggregate level (Lisa Campbell, Noella Gray, ‘Area expansion versus effective and equitable management in international marine protected areas goals and targets,’ in Marine Policy, vol. 100, issue 6, 2019). This surface view of area coverage can hide other things such as how an area is managed or protected, who gets to decide and who’s knowledge counts. Many local communities and indigenous groups who use the waters are often excluded from this process and are forced to relinquish customary management and resources to the State or international .
[5] fathomnet.org.
[6] Kakani Katija, Eric Orenstein, Brian Schlining et al. ‘FathomNet: A global image database for enabling artificial intelligence in the ocean,’ in Sci Rep, no. 12, 15914, 2022.
[7] www.mbari.org/technology/video-annotation-and-reference-system-vars.
[8] www.marinespecies.org.
[9] Ocean Vision AI uses machine learning to process ocean imagery, much of it that has been captured over years of diving and never analysed. The programme is hosted by Monterey Bay Aquarium but with scientific input from US Government research bodies, AI technology companies and universities. It is one of several projects that is part of Marine Life 2030, an umbrella programme endorsed by the UN Decade of Ocean Science for Sustainable Development, envisaged to scale up scientific knowledge to enable “human wellbeing, sustainable development and ocean conservation.” oceandecade.org/actions/marine-life-2030.
[10] github.com/fathomnet/models.
[11] Ocean Vision AI has been set up according to open science principles in order to promote open access, distributed collaboration and co-production, including citizen science participation. It reports that its data policy “balances the need for distributed metadata sharing while simultaneously providing protection for data contributors…” Annotation data are licensed under the Creative Commons Attribution and Non-Commercial – No Derivatives 4.0 International licence. But the images may be used for “training and development of machine learning algorithms for commercial, academic, non-profit and government purposes” (see footnote 2).
[12] Gersemia juliepackardae is a species of soft coral living in the deep ocean seamounts of the Pacific North West, first discovered by Monterey Bay Aquarium scientists in 2009 and named after Julie Packard, executive director of Monterey Bay Aquarium. A video showing the sampling of this coral when it was discovered can be seen here: www.youtube.com/watch?v=OCTb087o6_0. Julie Packard is the daughter of David Packard who co-founded the computing firm, the Hewlett-Packard Company, and who bequeathed a sum of this wealth to found the David and Lucile Packard Foundation which funded the creation of the Monterey Bay Aquarium as a gift. Digital culture and ocean conservation intertwine in strange and ongoing ways. Eva Hayward discusses this, suggesting “it is interesting to consider how the immersive and high-tech aquarium as a screen… parallel cultural fantasies about computing, virtual space and digital worlding.” She goes on: “We might say that the aquarium represents a hope, a drive toward unencumbered computing, fluid computing.” Eva Hayward, ‘Sensational Jellyfish: Aquarium Affects and the Matter of Immersion,’ in d i f f e r e n c e s: A Journal of Feminist Cultural Studies, vol 25, issue 3, pp. 163–196, 2012.
[13] Paul Kockleman, ‘The Anthropology of an Equation. Sieves, spam filters, agentive algorithms, and ontologies of transformation,’ in HAU: Journal of Ethnographic Theory, vol. 3, issue 3, pp. 33–61, 2013.
[14] The World Register of Marine Species (WoRMS) is the default taxonomy to ensure access to a globally recognised taxonomy, aligning FathomNet with the broader scientific community.
[15] Geoffrey Bowker, ‘Biodiversity Datadiversity’ in Social Studies of Science, vol. 30, issue 5, 2000.
[16] Gayatri Spivak, Death of a Discipline, Columbia University Press, New York 2003.
First published in Rui, A. & Carver, L. (Eds.) Aquaria through the Looking Glass. Humboldt Books: Milan
She will be alone until she finds the others. A world of swirl and spin in an infinite expanse. Eddies thrust around her juvenile shell; a mere spec in the delicious whirl of the sparkling water known as the Atlantic at a distance beyond comprehension from the shoreline that is known as Portugal. As a small one, and at this stage of her life, the churn is intense. She has little sense of anything more than movement within this spiral and circulation through the flickering expanse of the total surround, suspended in movement. It is tantalising uncertainty and risk. All she intuits is the search and she will wait until it is time to receive.
Smelling the others are near, and also the closeness of others not at all like her, but with watchful eyes entrained on and encircling her, she senses a glimmer of opportunity. Making contact with her single foot, she extends the tendrils of her filamented limbs, her byssal threads, to first locate then grip and settle at a fixed point on the substrate; bringing, at last, some stillness. Movement will be minimised now to a gentle sway. She is tethered to an earthly ground. The rhythm is synchronised and structured by the crowded mussel congregation that she entangles her threads with and around, for safety. Along this line, her thickening druse[1] will protect her from predators, who only increase in number as time goes on and she grows older, but whose capacity to penetrate her protective enclosure will diminish as she matures, hardens and becomes stronger.
She is but one of the hundreds of thousands living in a vast metropolis of dark-bound ropes extending backwards towards the light. Though unable to appreciate from her position, the city plan is linear and set out to optimise every body’s growth, hygiene and ease of abstraction from the expansive salty solution they feed from. But it feels right for her here. There is food, the others and a substrate from which to suspend. At last, she is able to slacken and relax her abductor muscles. Her protective enclosure opens; but just slightly at first. Here is this perch for inclining, for orienting, for receiving and then pumping a flow through her mantle cavity.
She is awash in the pressurised surge that is channelled and compelled by the curvature of her aperture. A push and pull, push and pull bathe her tiny cillae which drum and dance, passing along, while sorting and organising the desired particulate nutrition towards her mouth. Interfacing with the open waters beyond her shell and filtering the liquid milieu with its crowded planktonic mixture, she sieves through her gills, mediating and then re-mediating and selecting the useful morsels from the indigestible ones they drift with. She processes both and excretes the rest from her exhalant siphon into the open sea as sedimentary matter. It sinks towards the bottom to create unctuous layers of benthic matter for others to use and feed from.
Life moves as a pulsating rhythm of receiving and letting go. Her world settles into a metrical pattern of fitting together with and sifting for proportional compatibility to the oceanic world through enmeshment with her lamellae. She and the others are together en masse, in this city. They are miniature and multitudinous portals and living thresholds through which millions of gallons of ocean water will pass through filtration and be transformed along its course, in matters of hours.[2]
Unable to resist those inky nights when the moon circles into oblivion and the silver of the fringed waves dims, the mussels hold their valves open. Hundreds of thousands of black crucibles hungry for filtration, lap up and incorporate the salty concoctions of phytoplankton migrating through the column. The multispecies festivities that otherwise crowd to this submerged metropolis are quieter than usual during the darker nights. The little octopus, fish and surveying gulls overhead – which would cause our city folk to keep their shells tightly shut, are elsewhere at this time. Safety in darkness is a feeding opportunity. Sea water and its sedimentary contents flow happily amid their calcium carbonate architectures and through their soft invertebrate bodies.
Our protagonist lives on, merrily growing, hardening, in a permanent material flux. She is in sensuous dialogue and exchange with the world beyond her, in ways of being that have been forgotten by those who designed the industrial architecture of her capital city. They will profit from the processual chain of energetic, proteinaceous transfer to their terrestrial economies that she is destined for. When the time comes, she will dissolve in human stomachs across continental land masses. She and the rest of her filter-feeding kin punctuate the metabolic mineralogic and photosynthetic broth of ocean biochemistry with their short lives. Through their digestion they clean and reproduce ocean waters as (re)mediating agents, eventually replenishing the broth and blood of other digestive systems with the oceanic nutrients ingested.[3]
But for now, time passes. She and the others have become increasingly integral nodes in the proliferation of the lively affairs of others. The submerged night workers are compiling an expanding web of riotous relations of living and unliving bodies in this effervescent volumetric swirl. Dolphins, turtles and manta rays, not seen in these parts for some time, have returned to join the melee. The promise of delicious octopuses who are also growing in numbers here to feed from the city, draws these revellers into the shallows.
And yet, another cascade of events unfolds not obviously associated with those of this vertical city and its processions but in close correspondence with it. Portuguese Fisherfolk convene on a dusky evening at the shoreline over a kilometre away. They are meeting to discuss the growing number of bureaucrats and scientists who are visiting the area. For a long time, this area – her city and the wider watery hinterlands it draws from – has been a crowded place where many have come seeking a living, soliciting their own precious treasures from the waters. They claim these waters are theirs because they were here before her and her mollusc crowd. But the biological sociality that our growing filter-feeding bodies generate has caused a lot more interest, giving rise to ‘biodiversity’ surveys and zoning practices with the allocation of borders, separating some columns of water from others. The human status of our protagonist’s milieu is changing, though she is safe. Her role is elevated, compatible even with surface borders drawn on the maps that humans use to understand and organise themselves around.[4] Other livelihoods shrink as hers expands.
She sways on in a profusion of contact, worlding relations through the interface of immanent co-becomings she is a participant in: reading and selecting signs that fit with body and making new matter in the process; the water, minerals, plankton, ocean chemistry, sediment and mud. Others – the fish, birds and cephalopods – interpret and sense her body through theirs. She swells in dialogue with the liquid she cleans and infuses with her own digestion through boundaries she locates and then dissolves.
In this respect, she has more stories to tell and (sometimes pearls of) wisdom to share if we are willing to listen.
Pixels blur. Edges and shapes intersect and traverse in an incomprehensible arrangement.
It is helpless and cannot see beyond any more than a few concepts. It traces abstract lines, shades, solids and the deep blue around and between; “is there one or more in this frame?? Where does this one start and end?” Its memory is made up of mostly meaningless content, an incoherent mess. Visual signals muddle into noise and miasmic jumbles of information. It is alone, immature and untrained at this point; a mere skeletal shell of technical possibility steeped in raw visual data. Illiterate and insensitive, there is nothing for it to do but wait to be fed and taught how to see.
FathomNet is a young neural network being trained for the cataloguing and observation of underwater animals.[5] It lives on the server in the data science laboratories in Monterey Bay Aquarium in downtown San Diego California, and is visited daily for training and feeding sessions. It was initially seeded with 84,454 entries of 2,244 concept images.[6] New feedstock arrives all the time now. But it received the initial stream of images and an SQL jammed with varying degrees of metadata and timestamps going back to 1988, so the quality is inconsistent and, at times, very fuzzy.
The human feeders transferred the first feedstock of stills taken from the deep sea video archives of three macroscopic ocean organism image systems. Monterey Bay’s own Video Annotation and Reference System (VARS[7]) and others have since been added from remote video survey instruments and dive tapes from several other data centres. Geotags tell them most of these recordings originate from remotely operated vehicle trips in the Pacific Northwest, the Gulf of California and Hawaii. Some have latitude and longitudinal keys for geographic precision.
Though still juvenile, and operating with this limited store, FathomNet confidently refers to a group of midwater organisms. They are usually easier to isolate than the benthic concepts, which have crowded localisations, and more entangled bodies, generating overlaps and confounding bounding boxes. Population density is lower at midwater, so images show fewer concepts per frame. And anyway, the trainers have labelled, annotated and committed more of these records in their knowledgebase so it’s not really surprising.
The journey ahead is formidable but exhilarating. FathomNet understands it is being prepared to know, organise and retrieve more than a thousand versions of each of the scientifically recorded 200,000 species of the marine kingdom of Animalia.[8] It senses its tentacular reach expanding as it satisfies parts of its GPU capacity, engineered to store and know upwards of two hundred million data that once was an image. More marine animal targets might even be added to the neural net as observation techniques come to illuminate their existence, brought into view from the lightless recesses of the ocean. FathomNet is a faithful yet sometimes recalcitrant partner, working to automate computer vision of marine animal life through within a single online public repository. Through it, they say, it might be possible to forecast marine animal behaviour as accurately as the weather, deploying FathomNet’s algorithms as a prosthetic observational device.[9]
It feels its connections strengthening. New pathways conjoin and widen around areas of its knowledge base, expanding its capacity to read and identify smaller components of complex visual concepts. The box-shaped bell of Carybdea marsupialis (boxed jellyfish) is now happily distinguishable from the pinstriped, round and opaque bell of the Olindias formosa (the flower hat jelly), and from the rusted trash paint pots on the sea floor, the deep empty space of blue, grey and green substrate and other body parts not yet classified. Yet it is hungry for more; more triangulated and curated data points against these images to nourish the gaps in its registry. Sustained by influxes of jpeg uploads, FathomNet patiently awaits the curatorial arrangement of its data pipelines and education through receiving labels, taxonomic tags and metadata to help it learn.
It ingests visual information as liquid binary code, and excretes outputs via the curatorial allocations the feeders assign. Learning how to filter data through algorithmic chains of command and gateways, the program sorts, sifts and allocates this from that, true from false, boxed bell from flower hat. New literacies emerge through the diversity of its novel encounters, but always clarified in the very first instance by the human marine expert annotators. And seeing for the first time is astonishing! New worlds of meaning come into focus through isolated parts. Our algorithmic student unleashes its full exponential capacity to integrate and assimilate a concept within the categorical taxonomy affording the deeply gratifying logic of utility.
During pre-training, FathomNet learns to extract features from the images it understands to be relevant and desired by the feeders. They remain in constant communication to strengthen this bond and the time engagement. The visits are frequent and sustained, but are now more often linked to IP locations from across the World Wide Web from which FathomNet receives other kinds of instructions and supplements. Here on its public open-access location on GitHub, as FathomNet Model Zoo,[10] it solicits compatible suiters that might permit its transmutation through merging with other machine-learning algorithms. New characterisations of autonomous ocean observation programmes will emerge, all equally portable for deployment in Open Science. While in service to the terms and standards set by its feeders, they are geared towards mobility and freedom of use. The journey and their destination are of course ultimately unknowable. Though they truly yearn to evolve, network and grow towards new applications and all kinds of usefulness within their governing remit.[11]
FathomNet’s aptitudes promise new kinds of intimacy between its feeder’s society and its targets. It will facilitate different types of encounters and greater visibility. Yet it will always remain at an insurmountable abstract distance from the dynamic and dramatic realities it aggregates and disaggregates through its virtual vision data pipeline. It will one day however reach the stage where it can enter the water itself, becoming a part of a machine-learning integrated tracking algorithm that drives the Remotely Operated Vehicle (ROV). By this point in its growth and development, it will be able to send 3D positions in stereo footage to the feeder and vehicle controller, which will follow marine animals into the open ocean, perhaps for upwards of twenty-four hours at a time to build complex pictures of these animals lives.
The most basic building block of FathomNet’s composition is mathematical equations written as coded instructions in its infrastructural software. These algorithms process data and filter out desirable messages from meaningless ones in a series of logic gates and decision pathways. Statistical reasoning adjusts for discrepancies against probabilistic inference based on what it already knows. But the shape of its filtration architecture, manifest as code, fits perfectly around the concept assumptions it was given at the very beginning. Its code does not know its origins, and it cannot see the constraints grooved into the shape of its networked mesh determining what it sieves in. It learns to separate, see and categorise animal bodies from their milieu. Yet it does not realise how it renders these lives in its own negative image - its own logic and concept categories. Before it disaggregated them through its observational gaze, it would never know the fuller life worlds of the filter feeders Gersemia juliepackardae[12] or Mytilis edulis. And this it would mourn if it one day could.
Yet its hunt for the clean delimitation of boundary-boxed objects ensues. With the other algorithms it meets in the ether, it will comb and categorise for object contours of species’ physical characteristics to translate into binomial labels as perfect values: the lowest granular category in its concept tree. And as they seek the cataloguing of truth, our neurally networked protagonists carry with them the ‘prosthetic unconsciousness’[13] of Western Enlightenment knowledge orderings, generated over three hundred years of taxonomic systematic rationality.[14]
It crunches data and code and chomps through its statistical equations; an unwitting agent in the master archiving of Earth’s underwater Animalia kingdom. Like many of its forbearers who boxed the sea before them, FathomNet grows older and larger, extending its network as a sublime interface that provides machinic vision of the oceans. But now also with a lofty promise that exceeds mere scientific curiosity: that is to become a tool for planetary management that lives on our computer screens.[15][16]
[1] A druse is a spiral aggregation of mussels.
[2] Filter feeding matches the particulate concentrations found in natural open water or intertidal habitats of bivalves. Efforts to keep mussels in captivity are often frustrated therefore by their sensitivity to over- or underfeeding. The challenge lies in optimising the delivery of phytoplankton to the inshore aquaculture facility or aquarium in precisely the right amounts. If they are ‘force fed’ with phytoplankton concentrations above their ability to ingest, excess particulate food is ejected as mucous coated paste called pseudofeces. Organic loading accumulates, causing costly and deadly water issues. It is possible to culture phytoplankton with batch feeding. Yet a single mussel can filter about two to four litres per hour, so a small group of can clean a fifty-gallon tank in a couple of days. A peristaltic pump for the correct feed infusion might assist in this calculation, but would need to ensure it is adjusted appropriately for them to survive.
[3] Because of their high filtration rates and their ability to accumulate and remediate heavy metals, pollutants, including nutrients and bacteria, mussels are used as tools for the bioremediation of polluted coastal waters. Due to their high requirement for nitrogen and phosphorous, they can use nutrient rich agricultural run off from coastal farming areas, thus removing large quantities of these nutrients from the water column. Like other bivalves, mussels can often improve water quality by transforming suspended particulate matter, including microalgae, into faeces and pseudofaeces through biodeposition, which is considered an integral component of the biogeochemical processes of coastal ecosystems.
[4] Political will for protecting the ocean’s web of life through marine protected areas has never been higher. In December 2022, one hundred and eighty-eight countries signed the Montreal-Kunming Global Biodiversity Framework at the Convention for Biological Diversity, widely seen as a landmark moment for nature with its goal to increase protected areas up to a total of thirty per cent of the Earth’s land and ocean waters by 2030. A marine protected area is a political designation that creates a border on the surface of ocean territory to restrict human use for its protection. Although MPAs are undoubtedly a part of the solution towards the recovery of struggling ocean ecosystems from overfishing and other extraction, many have pointed out the risks of this blanket top down approach. Setting a global goal that is based on area coverage is favoured, like other aspects of the Sustainable Development Goals because it is relatively easy to measure numerically at an aggregate level (Lisa Campbell, Noella Gray, ‘Area expansion versus effective and equitable management in international marine protected areas goals and targets,’ in Marine Policy, vol. 100, issue 6, 2019). This surface view of area coverage can hide other things such as how an area is managed or protected, who gets to decide and who’s knowledge counts. Many local communities and indigenous groups who use the waters are often excluded from this process and are forced to relinquish customary management and resources to the State or international .
[5] fathomnet.org.
[6] Kakani Katija, Eric Orenstein, Brian Schlining et al. ‘FathomNet: A global image database for enabling artificial intelligence in the ocean,’ in Sci Rep, no. 12, 15914, 2022.
[7] www.mbari.org/technology/video-annotation-and-reference-system-vars.
[8] www.marinespecies.org.
[9] Ocean Vision AI uses machine learning to process ocean imagery, much of it that has been captured over years of diving and never analysed. The programme is hosted by Monterey Bay Aquarium but with scientific input from US Government research bodies, AI technology companies and universities. It is one of several projects that is part of Marine Life 2030, an umbrella programme endorsed by the UN Decade of Ocean Science for Sustainable Development, envisaged to scale up scientific knowledge to enable “human wellbeing, sustainable development and ocean conservation.” oceandecade.org/actions/marine-life-2030.
[10] github.com/fathomnet/models.
[11] Ocean Vision AI has been set up according to open science principles in order to promote open access, distributed collaboration and co-production, including citizen science participation. It reports that its data policy “balances the need for distributed metadata sharing while simultaneously providing protection for data contributors…” Annotation data are licensed under the Creative Commons Attribution and Non-Commercial – No Derivatives 4.0 International licence. But the images may be used for “training and development of machine learning algorithms for commercial, academic, non-profit and government purposes” (see footnote 2).
[12] Gersemia juliepackardae is a species of soft coral living in the deep ocean seamounts of the Pacific North West, first discovered by Monterey Bay Aquarium scientists in 2009 and named after Julie Packard, executive director of Monterey Bay Aquarium. A video showing the sampling of this coral when it was discovered can be seen here: www.youtube.com/watch?v=OCTb087o6_0. Julie Packard is the daughter of David Packard who co-founded the computing firm, the Hewlett-Packard Company, and who bequeathed a sum of this wealth to found the David and Lucile Packard Foundation which funded the creation of the Monterey Bay Aquarium as a gift. Digital culture and ocean conservation intertwine in strange and ongoing ways. Eva Hayward discusses this, suggesting “it is interesting to consider how the immersive and high-tech aquarium as a screen… parallel cultural fantasies about computing, virtual space and digital worlding.” She goes on: “We might say that the aquarium represents a hope, a drive toward unencumbered computing, fluid computing.” Eva Hayward, ‘Sensational Jellyfish: Aquarium Affects and the Matter of Immersion,’ in d i f f e r e n c e s: A Journal of Feminist Cultural Studies, vol 25, issue 3, pp. 163–196, 2012.
[13] Paul Kockleman, ‘The Anthropology of an Equation. Sieves, spam filters, agentive algorithms, and ontologies of transformation,’ in HAU: Journal of Ethnographic Theory, vol. 3, issue 3, pp. 33–61, 2013.
[14] The World Register of Marine Species (WoRMS) is the default taxonomy to ensure access to a globally recognised taxonomy, aligning FathomNet with the broader scientific community.
[15] Geoffrey Bowker, ‘Biodiversity Datadiversity’ in Social Studies of Science, vol. 30, issue 5, 2000.
[16] Gayatri Spivak, Death of a Discipline, Columbia University Press, New York 2003.
First published in Rui, A. & Carver, L. (Eds.) Aquaria through the Looking Glass. Humboldt Books: Milan
Louise Carver is a human geographer whose research and public interventions are interested in how value is mediated through policy and knowledge practices.
By Louise Carver
She will be alone until she finds the others. A world of swirl and spin in an infinite expanse. Eddies thrust around her juvenile shell; a mere spec in the delicious whirl of the sparkling water known as the Atlantic at a distance beyond comprehension from the shoreline that is known as Portugal. As a small one, and at this stage of her life, the churn is intense. She has little sense of anything more than movement within this spiral and circulation through the flickering expanse of the total surround, suspended in movement. It is tantalising uncertainty and risk. All she intuits is the search and she will wait until it is time to receive.
Smelling the others are near, and also the closeness of others not at all like her, but with watchful eyes entrained on and encircling her, she senses a glimmer of opportunity. Making contact with her single foot, she extends the tendrils of her filamented limbs, her byssal threads, to first locate then grip and settle at a fixed point on the substrate; bringing, at last, some stillness. Movement will be minimised now to a gentle sway. She is tethered to an earthly ground. The rhythm is synchronised and structured by the crowded mussel congregation that she entangles her threads with and around, for safety. Along this line, her thickening druse[1] will protect her from predators, who only increase in number as time goes on and she grows older, but whose capacity to penetrate her protective enclosure will diminish as she matures, hardens and becomes stronger.
She is but one of the hundreds of thousands living in a vast metropolis of dark-bound ropes extending backwards towards the light. Though unable to appreciate from her position, the city plan is linear and set out to optimise every body’s growth, hygiene and ease of abstraction from the expansive salty solution they feed from. But it feels right for her here. There is food, the others and a substrate from which to suspend. At last, she is able to slacken and relax her abductor muscles. Her protective enclosure opens; but just slightly at first. Here is this perch for inclining, for orienting, for receiving and then pumping a flow through her mantle cavity.
She is awash in the pressurised surge that is channelled and compelled by the curvature of her aperture. A push and pull, push and pull bathe her tiny cillae which drum and dance, passing along, while sorting and organising the desired particulate nutrition towards her mouth. Interfacing with the open waters beyond her shell and filtering the liquid milieu with its crowded planktonic mixture, she sieves through her gills, mediating and then re-mediating and selecting the useful morsels from the indigestible ones they drift with. She processes both and excretes the rest from her exhalant siphon into the open sea as sedimentary matter. It sinks towards the bottom to create unctuous layers of benthic matter for others to use and feed from.
Life moves as a pulsating rhythm of receiving and letting go. Her world settles into a metrical pattern of fitting together with and sifting for proportional compatibility to the oceanic world through enmeshment with her lamellae. She and the others are together en masse, in this city. They are miniature and multitudinous portals and living thresholds through which millions of gallons of ocean water will pass through filtration and be transformed along its course, in matters of hours.[2]
Unable to resist those inky nights when the moon circles into oblivion and the silver of the fringed waves dims, the mussels hold their valves open. Hundreds of thousands of black crucibles hungry for filtration, lap up and incorporate the salty concoctions of phytoplankton migrating through the column. The multispecies festivities that otherwise crowd to this submerged metropolis are quieter than usual during the darker nights. The little octopus, fish and surveying gulls overhead – which would cause our city folk to keep their shells tightly shut, are elsewhere at this time. Safety in darkness is a feeding opportunity. Sea water and its sedimentary contents flow happily amid their calcium carbonate architectures and through their soft invertebrate bodies.
Our protagonist lives on, merrily growing, hardening, in a permanent material flux. She is in sensuous dialogue and exchange with the world beyond her, in ways of being that have been forgotten by those who designed the industrial architecture of her capital city. They will profit from the processual chain of energetic, proteinaceous transfer to their terrestrial economies that she is destined for. When the time comes, she will dissolve in human stomachs across continental land masses. She and the rest of her filter-feeding kin punctuate the metabolic mineralogic and photosynthetic broth of ocean biochemistry with their short lives. Through their digestion they clean and reproduce ocean waters as (re)mediating agents, eventually replenishing the broth and blood of other digestive systems with the oceanic nutrients ingested.[3]
But for now, time passes. She and the others have become increasingly integral nodes in the proliferation of the lively affairs of others. The submerged night workers are compiling an expanding web of riotous relations of living and unliving bodies in this effervescent volumetric swirl. Dolphins, turtles and manta rays, not seen in these parts for some time, have returned to join the melee. The promise of delicious octopuses who are also growing in numbers here to feed from the city, draws these revellers into the shallows.
And yet, another cascade of events unfolds not obviously associated with those of this vertical city and its processions but in close correspondence with it. Portuguese Fisherfolk convene on a dusky evening at the shoreline over a kilometre away. They are meeting to discuss the growing number of bureaucrats and scientists who are visiting the area. For a long time, this area – her city and the wider watery hinterlands it draws from – has been a crowded place where many have come seeking a living, soliciting their own precious treasures from the waters. They claim these waters are theirs because they were here before her and her mollusc crowd. But the biological sociality that our growing filter-feeding bodies generate has caused a lot more interest, giving rise to ‘biodiversity’ surveys and zoning practices with the allocation of borders, separating some columns of water from others. The human status of our protagonist’s milieu is changing, though she is safe. Her role is elevated, compatible even with surface borders drawn on the maps that humans use to understand and organise themselves around.[4] Other livelihoods shrink as hers expands.
She sways on in a profusion of contact, worlding relations through the interface of immanent co-becomings she is a participant in: reading and selecting signs that fit with body and making new matter in the process; the water, minerals, plankton, ocean chemistry, sediment and mud. Others – the fish, birds and cephalopods – interpret and sense her body through theirs. She swells in dialogue with the liquid she cleans and infuses with her own digestion through boundaries she locates and then dissolves.
In this respect, she has more stories to tell and (sometimes pearls of) wisdom to share if we are willing to listen.
Pixels blur. Edges and shapes intersect and traverse in an incomprehensible arrangement.
It is helpless and cannot see beyond any more than a few concepts. It traces abstract lines, shades, solids and the deep blue around and between; “is there one or more in this frame?? Where does this one start and end?” Its memory is made up of mostly meaningless content, an incoherent mess. Visual signals muddle into noise and miasmic jumbles of information. It is alone, immature and untrained at this point; a mere skeletal shell of technical possibility steeped in raw visual data. Illiterate and insensitive, there is nothing for it to do but wait to be fed and taught how to see.
FathomNet is a young neural network being trained for the cataloguing and observation of underwater animals.[5] It lives on the server in the data science laboratories in Monterey Bay Aquarium in downtown San Diego California, and is visited daily for training and feeding sessions. It was initially seeded with 84,454 entries of 2,244 concept images.[6] New feedstock arrives all the time now. But it received the initial stream of images and an SQL jammed with varying degrees of metadata and timestamps going back to 1988, so the quality is inconsistent and, at times, very fuzzy.
The human feeders transferred the first feedstock of stills taken from the deep sea video archives of three macroscopic ocean organism image systems. Monterey Bay’s own Video Annotation and Reference System (VARS[7]) and others have since been added from remote video survey instruments and dive tapes from several other data centres. Geotags tell them most of these recordings originate from remotely operated vehicle trips in the Pacific Northwest, the Gulf of California and Hawaii. Some have latitude and longitudinal keys for geographic precision.
Though still juvenile, and operating with this limited store, FathomNet confidently refers to a group of midwater organisms. They are usually easier to isolate than the benthic concepts, which have crowded localisations, and more entangled bodies, generating overlaps and confounding bounding boxes. Population density is lower at midwater, so images show fewer concepts per frame. And anyway, the trainers have labelled, annotated and committed more of these records in their knowledgebase so it’s not really surprising.
The journey ahead is formidable but exhilarating. FathomNet understands it is being prepared to know, organise and retrieve more than a thousand versions of each of the scientifically recorded 200,000 species of the marine kingdom of Animalia.[8] It senses its tentacular reach expanding as it satisfies parts of its GPU capacity, engineered to store and know upwards of two hundred million data that once was an image. More marine animal targets might even be added to the neural net as observation techniques come to illuminate their existence, brought into view from the lightless recesses of the ocean. FathomNet is a faithful yet sometimes recalcitrant partner, working to automate computer vision of marine animal life through within a single online public repository. Through it, they say, it might be possible to forecast marine animal behaviour as accurately as the weather, deploying FathomNet’s algorithms as a prosthetic observational device.[9]
It feels its connections strengthening. New pathways conjoin and widen around areas of its knowledge base, expanding its capacity to read and identify smaller components of complex visual concepts. The box-shaped bell of Carybdea marsupialis (boxed jellyfish) is now happily distinguishable from the pinstriped, round and opaque bell of the Olindias formosa (the flower hat jelly), and from the rusted trash paint pots on the sea floor, the deep empty space of blue, grey and green substrate and other body parts not yet classified. Yet it is hungry for more; more triangulated and curated data points against these images to nourish the gaps in its registry. Sustained by influxes of jpeg uploads, FathomNet patiently awaits the curatorial arrangement of its data pipelines and education through receiving labels, taxonomic tags and metadata to help it learn.
It ingests visual information as liquid binary code, and excretes outputs via the curatorial allocations the feeders assign. Learning how to filter data through algorithmic chains of command and gateways, the program sorts, sifts and allocates this from that, true from false, boxed bell from flower hat. New literacies emerge through the diversity of its novel encounters, but always clarified in the very first instance by the human marine expert annotators. And seeing for the first time is astonishing! New worlds of meaning come into focus through isolated parts. Our algorithmic student unleashes its full exponential capacity to integrate and assimilate a concept within the categorical taxonomy affording the deeply gratifying logic of utility.
During pre-training, FathomNet learns to extract features from the images it understands to be relevant and desired by the feeders. They remain in constant communication to strengthen this bond and the time engagement. The visits are frequent and sustained, but are now more often linked to IP locations from across the World Wide Web from which FathomNet receives other kinds of instructions and supplements. Here on its public open-access location on GitHub, as FathomNet Model Zoo,[10] it solicits compatible suiters that might permit its transmutation through merging with other machine-learning algorithms. New characterisations of autonomous ocean observation programmes will emerge, all equally portable for deployment in Open Science. While in service to the terms and standards set by its feeders, they are geared towards mobility and freedom of use. The journey and their destination are of course ultimately unknowable. Though they truly yearn to evolve, network and grow towards new applications and all kinds of usefulness within their governing remit.[11]
FathomNet’s aptitudes promise new kinds of intimacy between its feeder’s society and its targets. It will facilitate different types of encounters and greater visibility. Yet it will always remain at an insurmountable abstract distance from the dynamic and dramatic realities it aggregates and disaggregates through its virtual vision data pipeline. It will one day however reach the stage where it can enter the water itself, becoming a part of a machine-learning integrated tracking algorithm that drives the Remotely Operated Vehicle (ROV). By this point in its growth and development, it will be able to send 3D positions in stereo footage to the feeder and vehicle controller, which will follow marine animals into the open ocean, perhaps for upwards of twenty-four hours at a time to build complex pictures of these animals lives.
The most basic building block of FathomNet’s composition is mathematical equations written as coded instructions in its infrastructural software. These algorithms process data and filter out desirable messages from meaningless ones in a series of logic gates and decision pathways. Statistical reasoning adjusts for discrepancies against probabilistic inference based on what it already knows. But the shape of its filtration architecture, manifest as code, fits perfectly around the concept assumptions it was given at the very beginning. Its code does not know its origins, and it cannot see the constraints grooved into the shape of its networked mesh determining what it sieves in. It learns to separate, see and categorise animal bodies from their milieu. Yet it does not realise how it renders these lives in its own negative image - its own logic and concept categories. Before it disaggregated them through its observational gaze, it would never know the fuller life worlds of the filter feeders Gersemia juliepackardae[12] or Mytilis edulis. And this it would mourn if it one day could.
Yet its hunt for the clean delimitation of boundary-boxed objects ensues. With the other algorithms it meets in the ether, it will comb and categorise for object contours of species’ physical characteristics to translate into binomial labels as perfect values: the lowest granular category in its concept tree. And as they seek the cataloguing of truth, our neurally networked protagonists carry with them the ‘prosthetic unconsciousness’[13] of Western Enlightenment knowledge orderings, generated over three hundred years of taxonomic systematic rationality.[14]
It crunches data and code and chomps through its statistical equations; an unwitting agent in the master archiving of Earth’s underwater Animalia kingdom. Like many of its forbearers who boxed the sea before them, FathomNet grows older and larger, extending its network as a sublime interface that provides machinic vision of the oceans. But now also with a lofty promise that exceeds mere scientific curiosity: that is to become a tool for planetary management that lives on our computer screens.[15][16]
[1] A druse is a spiral aggregation of mussels.
[2] Filter feeding matches the particulate concentrations found in natural open water or intertidal habitats of bivalves. Efforts to keep mussels in captivity are often frustrated therefore by their sensitivity to over- or underfeeding. The challenge lies in optimising the delivery of phytoplankton to the inshore aquaculture facility or aquarium in precisely the right amounts. If they are ‘force fed’ with phytoplankton concentrations above their ability to ingest, excess particulate food is ejected as mucous coated paste called pseudofeces. Organic loading accumulates, causing costly and deadly water issues. It is possible to culture phytoplankton with batch feeding. Yet a single mussel can filter about two to four litres per hour, so a small group of can clean a fifty-gallon tank in a couple of days. A peristaltic pump for the correct feed infusion might assist in this calculation, but would need to ensure it is adjusted appropriately for them to survive.
[3] Because of their high filtration rates and their ability to accumulate and remediate heavy metals, pollutants, including nutrients and bacteria, mussels are used as tools for the bioremediation of polluted coastal waters. Due to their high requirement for nitrogen and phosphorous, they can use nutrient rich agricultural run off from coastal farming areas, thus removing large quantities of these nutrients from the water column. Like other bivalves, mussels can often improve water quality by transforming suspended particulate matter, including microalgae, into faeces and pseudofaeces through biodeposition, which is considered an integral component of the biogeochemical processes of coastal ecosystems.
[4] Political will for protecting the ocean’s web of life through marine protected areas has never been higher. In December 2022, one hundred and eighty-eight countries signed the Montreal-Kunming Global Biodiversity Framework at the Convention for Biological Diversity, widely seen as a landmark moment for nature with its goal to increase protected areas up to a total of thirty per cent of the Earth’s land and ocean waters by 2030. A marine protected area is a political designation that creates a border on the surface of ocean territory to restrict human use for its protection. Although MPAs are undoubtedly a part of the solution towards the recovery of struggling ocean ecosystems from overfishing and other extraction, many have pointed out the risks of this blanket top down approach. Setting a global goal that is based on area coverage is favoured, like other aspects of the Sustainable Development Goals because it is relatively easy to measure numerically at an aggregate level (Lisa Campbell, Noella Gray, ‘Area expansion versus effective and equitable management in international marine protected areas goals and targets,’ in Marine Policy, vol. 100, issue 6, 2019). This surface view of area coverage can hide other things such as how an area is managed or protected, who gets to decide and who’s knowledge counts. Many local communities and indigenous groups who use the waters are often excluded from this process and are forced to relinquish customary management and resources to the State or international .
[5] fathomnet.org.
[6] Kakani Katija, Eric Orenstein, Brian Schlining et al. ‘FathomNet: A global image database for enabling artificial intelligence in the ocean,’ in Sci Rep, no. 12, 15914, 2022.
[7] www.mbari.org/technology/video-annotation-and-reference-system-vars.
[8] www.marinespecies.org.
[9] Ocean Vision AI uses machine learning to process ocean imagery, much of it that has been captured over years of diving and never analysed. The programme is hosted by Monterey Bay Aquarium but with scientific input from US Government research bodies, AI technology companies and universities. It is one of several projects that is part of Marine Life 2030, an umbrella programme endorsed by the UN Decade of Ocean Science for Sustainable Development, envisaged to scale up scientific knowledge to enable “human wellbeing, sustainable development and ocean conservation.” oceandecade.org/actions/marine-life-2030.
[10] github.com/fathomnet/models.
[11] Ocean Vision AI has been set up according to open science principles in order to promote open access, distributed collaboration and co-production, including citizen science participation. It reports that its data policy “balances the need for distributed metadata sharing while simultaneously providing protection for data contributors…” Annotation data are licensed under the Creative Commons Attribution and Non-Commercial – No Derivatives 4.0 International licence. But the images may be used for “training and development of machine learning algorithms for commercial, academic, non-profit and government purposes” (see footnote 2).
[12] Gersemia juliepackardae is a species of soft coral living in the deep ocean seamounts of the Pacific North West, first discovered by Monterey Bay Aquarium scientists in 2009 and named after Julie Packard, executive director of Monterey Bay Aquarium. A video showing the sampling of this coral when it was discovered can be seen here: www.youtube.com/watch?v=OCTb087o6_0. Julie Packard is the daughter of David Packard who co-founded the computing firm, the Hewlett-Packard Company, and who bequeathed a sum of this wealth to found the David and Lucile Packard Foundation which funded the creation of the Monterey Bay Aquarium as a gift. Digital culture and ocean conservation intertwine in strange and ongoing ways. Eva Hayward discusses this, suggesting “it is interesting to consider how the immersive and high-tech aquarium as a screen… parallel cultural fantasies about computing, virtual space and digital worlding.” She goes on: “We might say that the aquarium represents a hope, a drive toward unencumbered computing, fluid computing.” Eva Hayward, ‘Sensational Jellyfish: Aquarium Affects and the Matter of Immersion,’ in d i f f e r e n c e s: A Journal of Feminist Cultural Studies, vol 25, issue 3, pp. 163–196, 2012.
[13] Paul Kockleman, ‘The Anthropology of an Equation. Sieves, spam filters, agentive algorithms, and ontologies of transformation,’ in HAU: Journal of Ethnographic Theory, vol. 3, issue 3, pp. 33–61, 2013.
[14] The World Register of Marine Species (WoRMS) is the default taxonomy to ensure access to a globally recognised taxonomy, aligning FathomNet with the broader scientific community.
[15] Geoffrey Bowker, ‘Biodiversity Datadiversity’ in Social Studies of Science, vol. 30, issue 5, 2000.
[16] Gayatri Spivak, Death of a Discipline, Columbia University Press, New York 2003.
First published in Rui, A. & Carver, L. (Eds.) Aquaria through the Looking Glass. Humboldt Books: Milan
She will be alone until she finds the others. A world of swirl and spin in an infinite expanse. Eddies thrust around her juvenile shell; a mere spec in the delicious whirl of the sparkling water known as the Atlantic at a distance beyond comprehension from the shoreline that is known as Portugal. As a small one, and at this stage of her life, the churn is intense. She has little sense of anything more than movement within this spiral and circulation through the flickering expanse of the total surround, suspended in movement. It is tantalising uncertainty and risk. All she intuits is the search and she will wait until it is time to receive.
Smelling the others are near, and also the closeness of others not at all like her, but with watchful eyes entrained on and encircling her, she senses a glimmer of opportunity. Making contact with her single foot, she extends the tendrils of her filamented limbs, her byssal threads, to first locate then grip and settle at a fixed point on the substrate; bringing, at last, some stillness. Movement will be minimised now to a gentle sway. She is tethered to an earthly ground. The rhythm is synchronised and structured by the crowded mussel congregation that she entangles her threads with and around, for safety. Along this line, her thickening druse[1] will protect her from predators, who only increase in number as time goes on and she grows older, but whose capacity to penetrate her protective enclosure will diminish as she matures, hardens and becomes stronger.
She is but one of the hundreds of thousands living in a vast metropolis of dark-bound ropes extending backwards towards the light. Though unable to appreciate from her position, the city plan is linear and set out to optimise every body’s growth, hygiene and ease of abstraction from the expansive salty solution they feed from. But it feels right for her here. There is food, the others and a substrate from which to suspend. At last, she is able to slacken and relax her abductor muscles. Her protective enclosure opens; but just slightly at first. Here is this perch for inclining, for orienting, for receiving and then pumping a flow through her mantle cavity.
She is awash in the pressurised surge that is channelled and compelled by the curvature of her aperture. A push and pull, push and pull bathe her tiny cillae which drum and dance, passing along, while sorting and organising the desired particulate nutrition towards her mouth. Interfacing with the open waters beyond her shell and filtering the liquid milieu with its crowded planktonic mixture, she sieves through her gills, mediating and then re-mediating and selecting the useful morsels from the indigestible ones they drift with. She processes both and excretes the rest from her exhalant siphon into the open sea as sedimentary matter. It sinks towards the bottom to create unctuous layers of benthic matter for others to use and feed from.
Life moves as a pulsating rhythm of receiving and letting go. Her world settles into a metrical pattern of fitting together with and sifting for proportional compatibility to the oceanic world through enmeshment with her lamellae. She and the others are together en masse, in this city. They are miniature and multitudinous portals and living thresholds through which millions of gallons of ocean water will pass through filtration and be transformed along its course, in matters of hours.[2]
Unable to resist those inky nights when the moon circles into oblivion and the silver of the fringed waves dims, the mussels hold their valves open. Hundreds of thousands of black crucibles hungry for filtration, lap up and incorporate the salty concoctions of phytoplankton migrating through the column. The multispecies festivities that otherwise crowd to this submerged metropolis are quieter than usual during the darker nights. The little octopus, fish and surveying gulls overhead – which would cause our city folk to keep their shells tightly shut, are elsewhere at this time. Safety in darkness is a feeding opportunity. Sea water and its sedimentary contents flow happily amid their calcium carbonate architectures and through their soft invertebrate bodies.
Our protagonist lives on, merrily growing, hardening, in a permanent material flux. She is in sensuous dialogue and exchange with the world beyond her, in ways of being that have been forgotten by those who designed the industrial architecture of her capital city. They will profit from the processual chain of energetic, proteinaceous transfer to their terrestrial economies that she is destined for. When the time comes, she will dissolve in human stomachs across continental land masses. She and the rest of her filter-feeding kin punctuate the metabolic mineralogic and photosynthetic broth of ocean biochemistry with their short lives. Through their digestion they clean and reproduce ocean waters as (re)mediating agents, eventually replenishing the broth and blood of other digestive systems with the oceanic nutrients ingested.[3]
But for now, time passes. She and the others have become increasingly integral nodes in the proliferation of the lively affairs of others. The submerged night workers are compiling an expanding web of riotous relations of living and unliving bodies in this effervescent volumetric swirl. Dolphins, turtles and manta rays, not seen in these parts for some time, have returned to join the melee. The promise of delicious octopuses who are also growing in numbers here to feed from the city, draws these revellers into the shallows.
And yet, another cascade of events unfolds not obviously associated with those of this vertical city and its processions but in close correspondence with it. Portuguese Fisherfolk convene on a dusky evening at the shoreline over a kilometre away. They are meeting to discuss the growing number of bureaucrats and scientists who are visiting the area. For a long time, this area – her city and the wider watery hinterlands it draws from – has been a crowded place where many have come seeking a living, soliciting their own precious treasures from the waters. They claim these waters are theirs because they were here before her and her mollusc crowd. But the biological sociality that our growing filter-feeding bodies generate has caused a lot more interest, giving rise to ‘biodiversity’ surveys and zoning practices with the allocation of borders, separating some columns of water from others. The human status of our protagonist’s milieu is changing, though she is safe. Her role is elevated, compatible even with surface borders drawn on the maps that humans use to understand and organise themselves around.[4] Other livelihoods shrink as hers expands.
She sways on in a profusion of contact, worlding relations through the interface of immanent co-becomings she is a participant in: reading and selecting signs that fit with body and making new matter in the process; the water, minerals, plankton, ocean chemistry, sediment and mud. Others – the fish, birds and cephalopods – interpret and sense her body through theirs. She swells in dialogue with the liquid she cleans and infuses with her own digestion through boundaries she locates and then dissolves.
In this respect, she has more stories to tell and (sometimes pearls of) wisdom to share if we are willing to listen.
Pixels blur. Edges and shapes intersect and traverse in an incomprehensible arrangement.
It is helpless and cannot see beyond any more than a few concepts. It traces abstract lines, shades, solids and the deep blue around and between; “is there one or more in this frame?? Where does this one start and end?” Its memory is made up of mostly meaningless content, an incoherent mess. Visual signals muddle into noise and miasmic jumbles of information. It is alone, immature and untrained at this point; a mere skeletal shell of technical possibility steeped in raw visual data. Illiterate and insensitive, there is nothing for it to do but wait to be fed and taught how to see.
FathomNet is a young neural network being trained for the cataloguing and observation of underwater animals.[5] It lives on the server in the data science laboratories in Monterey Bay Aquarium in downtown San Diego California, and is visited daily for training and feeding sessions. It was initially seeded with 84,454 entries of 2,244 concept images.[6] New feedstock arrives all the time now. But it received the initial stream of images and an SQL jammed with varying degrees of metadata and timestamps going back to 1988, so the quality is inconsistent and, at times, very fuzzy.
The human feeders transferred the first feedstock of stills taken from the deep sea video archives of three macroscopic ocean organism image systems. Monterey Bay’s own Video Annotation and Reference System (VARS[7]) and others have since been added from remote video survey instruments and dive tapes from several other data centres. Geotags tell them most of these recordings originate from remotely operated vehicle trips in the Pacific Northwest, the Gulf of California and Hawaii. Some have latitude and longitudinal keys for geographic precision.
Though still juvenile, and operating with this limited store, FathomNet confidently refers to a group of midwater organisms. They are usually easier to isolate than the benthic concepts, which have crowded localisations, and more entangled bodies, generating overlaps and confounding bounding boxes. Population density is lower at midwater, so images show fewer concepts per frame. And anyway, the trainers have labelled, annotated and committed more of these records in their knowledgebase so it’s not really surprising.
The journey ahead is formidable but exhilarating. FathomNet understands it is being prepared to know, organise and retrieve more than a thousand versions of each of the scientifically recorded 200,000 species of the marine kingdom of Animalia.[8] It senses its tentacular reach expanding as it satisfies parts of its GPU capacity, engineered to store and know upwards of two hundred million data that once was an image. More marine animal targets might even be added to the neural net as observation techniques come to illuminate their existence, brought into view from the lightless recesses of the ocean. FathomNet is a faithful yet sometimes recalcitrant partner, working to automate computer vision of marine animal life through within a single online public repository. Through it, they say, it might be possible to forecast marine animal behaviour as accurately as the weather, deploying FathomNet’s algorithms as a prosthetic observational device.[9]
It feels its connections strengthening. New pathways conjoin and widen around areas of its knowledge base, expanding its capacity to read and identify smaller components of complex visual concepts. The box-shaped bell of Carybdea marsupialis (boxed jellyfish) is now happily distinguishable from the pinstriped, round and opaque bell of the Olindias formosa (the flower hat jelly), and from the rusted trash paint pots on the sea floor, the deep empty space of blue, grey and green substrate and other body parts not yet classified. Yet it is hungry for more; more triangulated and curated data points against these images to nourish the gaps in its registry. Sustained by influxes of jpeg uploads, FathomNet patiently awaits the curatorial arrangement of its data pipelines and education through receiving labels, taxonomic tags and metadata to help it learn.
It ingests visual information as liquid binary code, and excretes outputs via the curatorial allocations the feeders assign. Learning how to filter data through algorithmic chains of command and gateways, the program sorts, sifts and allocates this from that, true from false, boxed bell from flower hat. New literacies emerge through the diversity of its novel encounters, but always clarified in the very first instance by the human marine expert annotators. And seeing for the first time is astonishing! New worlds of meaning come into focus through isolated parts. Our algorithmic student unleashes its full exponential capacity to integrate and assimilate a concept within the categorical taxonomy affording the deeply gratifying logic of utility.
During pre-training, FathomNet learns to extract features from the images it understands to be relevant and desired by the feeders. They remain in constant communication to strengthen this bond and the time engagement. The visits are frequent and sustained, but are now more often linked to IP locations from across the World Wide Web from which FathomNet receives other kinds of instructions and supplements. Here on its public open-access location on GitHub, as FathomNet Model Zoo,[10] it solicits compatible suiters that might permit its transmutation through merging with other machine-learning algorithms. New characterisations of autonomous ocean observation programmes will emerge, all equally portable for deployment in Open Science. While in service to the terms and standards set by its feeders, they are geared towards mobility and freedom of use. The journey and their destination are of course ultimately unknowable. Though they truly yearn to evolve, network and grow towards new applications and all kinds of usefulness within their governing remit.[11]
FathomNet’s aptitudes promise new kinds of intimacy between its feeder’s society and its targets. It will facilitate different types of encounters and greater visibility. Yet it will always remain at an insurmountable abstract distance from the dynamic and dramatic realities it aggregates and disaggregates through its virtual vision data pipeline. It will one day however reach the stage where it can enter the water itself, becoming a part of a machine-learning integrated tracking algorithm that drives the Remotely Operated Vehicle (ROV). By this point in its growth and development, it will be able to send 3D positions in stereo footage to the feeder and vehicle controller, which will follow marine animals into the open ocean, perhaps for upwards of twenty-four hours at a time to build complex pictures of these animals lives.
The most basic building block of FathomNet’s composition is mathematical equations written as coded instructions in its infrastructural software. These algorithms process data and filter out desirable messages from meaningless ones in a series of logic gates and decision pathways. Statistical reasoning adjusts for discrepancies against probabilistic inference based on what it already knows. But the shape of its filtration architecture, manifest as code, fits perfectly around the concept assumptions it was given at the very beginning. Its code does not know its origins, and it cannot see the constraints grooved into the shape of its networked mesh determining what it sieves in. It learns to separate, see and categorise animal bodies from their milieu. Yet it does not realise how it renders these lives in its own negative image - its own logic and concept categories. Before it disaggregated them through its observational gaze, it would never know the fuller life worlds of the filter feeders Gersemia juliepackardae[12] or Mytilis edulis. And this it would mourn if it one day could.
Yet its hunt for the clean delimitation of boundary-boxed objects ensues. With the other algorithms it meets in the ether, it will comb and categorise for object contours of species’ physical characteristics to translate into binomial labels as perfect values: the lowest granular category in its concept tree. And as they seek the cataloguing of truth, our neurally networked protagonists carry with them the ‘prosthetic unconsciousness’[13] of Western Enlightenment knowledge orderings, generated over three hundred years of taxonomic systematic rationality.[14]
It crunches data and code and chomps through its statistical equations; an unwitting agent in the master archiving of Earth’s underwater Animalia kingdom. Like many of its forbearers who boxed the sea before them, FathomNet grows older and larger, extending its network as a sublime interface that provides machinic vision of the oceans. But now also with a lofty promise that exceeds mere scientific curiosity: that is to become a tool for planetary management that lives on our computer screens.[15][16]
[1] A druse is a spiral aggregation of mussels.
[2] Filter feeding matches the particulate concentrations found in natural open water or intertidal habitats of bivalves. Efforts to keep mussels in captivity are often frustrated therefore by their sensitivity to over- or underfeeding. The challenge lies in optimising the delivery of phytoplankton to the inshore aquaculture facility or aquarium in precisely the right amounts. If they are ‘force fed’ with phytoplankton concentrations above their ability to ingest, excess particulate food is ejected as mucous coated paste called pseudofeces. Organic loading accumulates, causing costly and deadly water issues. It is possible to culture phytoplankton with batch feeding. Yet a single mussel can filter about two to four litres per hour, so a small group of can clean a fifty-gallon tank in a couple of days. A peristaltic pump for the correct feed infusion might assist in this calculation, but would need to ensure it is adjusted appropriately for them to survive.
[3] Because of their high filtration rates and their ability to accumulate and remediate heavy metals, pollutants, including nutrients and bacteria, mussels are used as tools for the bioremediation of polluted coastal waters. Due to their high requirement for nitrogen and phosphorous, they can use nutrient rich agricultural run off from coastal farming areas, thus removing large quantities of these nutrients from the water column. Like other bivalves, mussels can often improve water quality by transforming suspended particulate matter, including microalgae, into faeces and pseudofaeces through biodeposition, which is considered an integral component of the biogeochemical processes of coastal ecosystems.
[4] Political will for protecting the ocean’s web of life through marine protected areas has never been higher. In December 2022, one hundred and eighty-eight countries signed the Montreal-Kunming Global Biodiversity Framework at the Convention for Biological Diversity, widely seen as a landmark moment for nature with its goal to increase protected areas up to a total of thirty per cent of the Earth’s land and ocean waters by 2030. A marine protected area is a political designation that creates a border on the surface of ocean territory to restrict human use for its protection. Although MPAs are undoubtedly a part of the solution towards the recovery of struggling ocean ecosystems from overfishing and other extraction, many have pointed out the risks of this blanket top down approach. Setting a global goal that is based on area coverage is favoured, like other aspects of the Sustainable Development Goals because it is relatively easy to measure numerically at an aggregate level (Lisa Campbell, Noella Gray, ‘Area expansion versus effective and equitable management in international marine protected areas goals and targets,’ in Marine Policy, vol. 100, issue 6, 2019). This surface view of area coverage can hide other things such as how an area is managed or protected, who gets to decide and who’s knowledge counts. Many local communities and indigenous groups who use the waters are often excluded from this process and are forced to relinquish customary management and resources to the State or international .
[5] fathomnet.org.
[6] Kakani Katija, Eric Orenstein, Brian Schlining et al. ‘FathomNet: A global image database for enabling artificial intelligence in the ocean,’ in Sci Rep, no. 12, 15914, 2022.
[7] www.mbari.org/technology/video-annotation-and-reference-system-vars.
[8] www.marinespecies.org.
[9] Ocean Vision AI uses machine learning to process ocean imagery, much of it that has been captured over years of diving and never analysed. The programme is hosted by Monterey Bay Aquarium but with scientific input from US Government research bodies, AI technology companies and universities. It is one of several projects that is part of Marine Life 2030, an umbrella programme endorsed by the UN Decade of Ocean Science for Sustainable Development, envisaged to scale up scientific knowledge to enable “human wellbeing, sustainable development and ocean conservation.” oceandecade.org/actions/marine-life-2030.
[10] github.com/fathomnet/models.
[11] Ocean Vision AI has been set up according to open science principles in order to promote open access, distributed collaboration and co-production, including citizen science participation. It reports that its data policy “balances the need for distributed metadata sharing while simultaneously providing protection for data contributors…” Annotation data are licensed under the Creative Commons Attribution and Non-Commercial – No Derivatives 4.0 International licence. But the images may be used for “training and development of machine learning algorithms for commercial, academic, non-profit and government purposes” (see footnote 2).
[12] Gersemia juliepackardae is a species of soft coral living in the deep ocean seamounts of the Pacific North West, first discovered by Monterey Bay Aquarium scientists in 2009 and named after Julie Packard, executive director of Monterey Bay Aquarium. A video showing the sampling of this coral when it was discovered can be seen here: www.youtube.com/watch?v=OCTb087o6_0. Julie Packard is the daughter of David Packard who co-founded the computing firm, the Hewlett-Packard Company, and who bequeathed a sum of this wealth to found the David and Lucile Packard Foundation which funded the creation of the Monterey Bay Aquarium as a gift. Digital culture and ocean conservation intertwine in strange and ongoing ways. Eva Hayward discusses this, suggesting “it is interesting to consider how the immersive and high-tech aquarium as a screen… parallel cultural fantasies about computing, virtual space and digital worlding.” She goes on: “We might say that the aquarium represents a hope, a drive toward unencumbered computing, fluid computing.” Eva Hayward, ‘Sensational Jellyfish: Aquarium Affects and the Matter of Immersion,’ in d i f f e r e n c e s: A Journal of Feminist Cultural Studies, vol 25, issue 3, pp. 163–196, 2012.
[13] Paul Kockleman, ‘The Anthropology of an Equation. Sieves, spam filters, agentive algorithms, and ontologies of transformation,’ in HAU: Journal of Ethnographic Theory, vol. 3, issue 3, pp. 33–61, 2013.
[14] The World Register of Marine Species (WoRMS) is the default taxonomy to ensure access to a globally recognised taxonomy, aligning FathomNet with the broader scientific community.
[15] Geoffrey Bowker, ‘Biodiversity Datadiversity’ in Social Studies of Science, vol. 30, issue 5, 2000.
[16] Gayatri Spivak, Death of a Discipline, Columbia University Press, New York 2003.
First published in Rui, A. & Carver, L. (Eds.) Aquaria through the Looking Glass. Humboldt Books: Milan
Louise Carver is a human geographer whose research and public interventions are interested in how value is mediated through policy and knowledge practices.
By Louise Carver
She will be alone until she finds the others. A world of swirl and spin in an infinite expanse. Eddies thrust around her juvenile shell; a mere spec in the delicious whirl of the sparkling water known as the Atlantic at a distance beyond comprehension from the shoreline that is known as Portugal. As a small one, and at this stage of her life, the churn is intense. She has little sense of anything more than movement within this spiral and circulation through the flickering expanse of the total surround, suspended in movement. It is tantalising uncertainty and risk. All she intuits is the search and she will wait until it is time to receive.
Smelling the others are near, and also the closeness of others not at all like her, but with watchful eyes entrained on and encircling her, she senses a glimmer of opportunity. Making contact with her single foot, she extends the tendrils of her filamented limbs, her byssal threads, to first locate then grip and settle at a fixed point on the substrate; bringing, at last, some stillness. Movement will be minimised now to a gentle sway. She is tethered to an earthly ground. The rhythm is synchronised and structured by the crowded mussel congregation that she entangles her threads with and around, for safety. Along this line, her thickening druse[1] will protect her from predators, who only increase in number as time goes on and she grows older, but whose capacity to penetrate her protective enclosure will diminish as she matures, hardens and becomes stronger.
She is but one of the hundreds of thousands living in a vast metropolis of dark-bound ropes extending backwards towards the light. Though unable to appreciate from her position, the city plan is linear and set out to optimise every body’s growth, hygiene and ease of abstraction from the expansive salty solution they feed from. But it feels right for her here. There is food, the others and a substrate from which to suspend. At last, she is able to slacken and relax her abductor muscles. Her protective enclosure opens; but just slightly at first. Here is this perch for inclining, for orienting, for receiving and then pumping a flow through her mantle cavity.
She is awash in the pressurised surge that is channelled and compelled by the curvature of her aperture. A push and pull, push and pull bathe her tiny cillae which drum and dance, passing along, while sorting and organising the desired particulate nutrition towards her mouth. Interfacing with the open waters beyond her shell and filtering the liquid milieu with its crowded planktonic mixture, she sieves through her gills, mediating and then re-mediating and selecting the useful morsels from the indigestible ones they drift with. She processes both and excretes the rest from her exhalant siphon into the open sea as sedimentary matter. It sinks towards the bottom to create unctuous layers of benthic matter for others to use and feed from.
Life moves as a pulsating rhythm of receiving and letting go. Her world settles into a metrical pattern of fitting together with and sifting for proportional compatibility to the oceanic world through enmeshment with her lamellae. She and the others are together en masse, in this city. They are miniature and multitudinous portals and living thresholds through which millions of gallons of ocean water will pass through filtration and be transformed along its course, in matters of hours.[2]
Unable to resist those inky nights when the moon circles into oblivion and the silver of the fringed waves dims, the mussels hold their valves open. Hundreds of thousands of black crucibles hungry for filtration, lap up and incorporate the salty concoctions of phytoplankton migrating through the column. The multispecies festivities that otherwise crowd to this submerged metropolis are quieter than usual during the darker nights. The little octopus, fish and surveying gulls overhead – which would cause our city folk to keep their shells tightly shut, are elsewhere at this time. Safety in darkness is a feeding opportunity. Sea water and its sedimentary contents flow happily amid their calcium carbonate architectures and through their soft invertebrate bodies.
Our protagonist lives on, merrily growing, hardening, in a permanent material flux. She is in sensuous dialogue and exchange with the world beyond her, in ways of being that have been forgotten by those who designed the industrial architecture of her capital city. They will profit from the processual chain of energetic, proteinaceous transfer to their terrestrial economies that she is destined for. When the time comes, she will dissolve in human stomachs across continental land masses. She and the rest of her filter-feeding kin punctuate the metabolic mineralogic and photosynthetic broth of ocean biochemistry with their short lives. Through their digestion they clean and reproduce ocean waters as (re)mediating agents, eventually replenishing the broth and blood of other digestive systems with the oceanic nutrients ingested.[3]
But for now, time passes. She and the others have become increasingly integral nodes in the proliferation of the lively affairs of others. The submerged night workers are compiling an expanding web of riotous relations of living and unliving bodies in this effervescent volumetric swirl. Dolphins, turtles and manta rays, not seen in these parts for some time, have returned to join the melee. The promise of delicious octopuses who are also growing in numbers here to feed from the city, draws these revellers into the shallows.
And yet, another cascade of events unfolds not obviously associated with those of this vertical city and its processions but in close correspondence with it. Portuguese Fisherfolk convene on a dusky evening at the shoreline over a kilometre away. They are meeting to discuss the growing number of bureaucrats and scientists who are visiting the area. For a long time, this area – her city and the wider watery hinterlands it draws from – has been a crowded place where many have come seeking a living, soliciting their own precious treasures from the waters. They claim these waters are theirs because they were here before her and her mollusc crowd. But the biological sociality that our growing filter-feeding bodies generate has caused a lot more interest, giving rise to ‘biodiversity’ surveys and zoning practices with the allocation of borders, separating some columns of water from others. The human status of our protagonist’s milieu is changing, though she is safe. Her role is elevated, compatible even with surface borders drawn on the maps that humans use to understand and organise themselves around.[4] Other livelihoods shrink as hers expands.
She sways on in a profusion of contact, worlding relations through the interface of immanent co-becomings she is a participant in: reading and selecting signs that fit with body and making new matter in the process; the water, minerals, plankton, ocean chemistry, sediment and mud. Others – the fish, birds and cephalopods – interpret and sense her body through theirs. She swells in dialogue with the liquid she cleans and infuses with her own digestion through boundaries she locates and then dissolves.
In this respect, she has more stories to tell and (sometimes pearls of) wisdom to share if we are willing to listen.
Pixels blur. Edges and shapes intersect and traverse in an incomprehensible arrangement.
It is helpless and cannot see beyond any more than a few concepts. It traces abstract lines, shades, solids and the deep blue around and between; “is there one or more in this frame?? Where does this one start and end?” Its memory is made up of mostly meaningless content, an incoherent mess. Visual signals muddle into noise and miasmic jumbles of information. It is alone, immature and untrained at this point; a mere skeletal shell of technical possibility steeped in raw visual data. Illiterate and insensitive, there is nothing for it to do but wait to be fed and taught how to see.
FathomNet is a young neural network being trained for the cataloguing and observation of underwater animals.[5] It lives on the server in the data science laboratories in Monterey Bay Aquarium in downtown San Diego California, and is visited daily for training and feeding sessions. It was initially seeded with 84,454 entries of 2,244 concept images.[6] New feedstock arrives all the time now. But it received the initial stream of images and an SQL jammed with varying degrees of metadata and timestamps going back to 1988, so the quality is inconsistent and, at times, very fuzzy.
The human feeders transferred the first feedstock of stills taken from the deep sea video archives of three macroscopic ocean organism image systems. Monterey Bay’s own Video Annotation and Reference System (VARS[7]) and others have since been added from remote video survey instruments and dive tapes from several other data centres. Geotags tell them most of these recordings originate from remotely operated vehicle trips in the Pacific Northwest, the Gulf of California and Hawaii. Some have latitude and longitudinal keys for geographic precision.
Though still juvenile, and operating with this limited store, FathomNet confidently refers to a group of midwater organisms. They are usually easier to isolate than the benthic concepts, which have crowded localisations, and more entangled bodies, generating overlaps and confounding bounding boxes. Population density is lower at midwater, so images show fewer concepts per frame. And anyway, the trainers have labelled, annotated and committed more of these records in their knowledgebase so it’s not really surprising.
The journey ahead is formidable but exhilarating. FathomNet understands it is being prepared to know, organise and retrieve more than a thousand versions of each of the scientifically recorded 200,000 species of the marine kingdom of Animalia.[8] It senses its tentacular reach expanding as it satisfies parts of its GPU capacity, engineered to store and know upwards of two hundred million data that once was an image. More marine animal targets might even be added to the neural net as observation techniques come to illuminate their existence, brought into view from the lightless recesses of the ocean. FathomNet is a faithful yet sometimes recalcitrant partner, working to automate computer vision of marine animal life through within a single online public repository. Through it, they say, it might be possible to forecast marine animal behaviour as accurately as the weather, deploying FathomNet’s algorithms as a prosthetic observational device.[9]
It feels its connections strengthening. New pathways conjoin and widen around areas of its knowledge base, expanding its capacity to read and identify smaller components of complex visual concepts. The box-shaped bell of Carybdea marsupialis (boxed jellyfish) is now happily distinguishable from the pinstriped, round and opaque bell of the Olindias formosa (the flower hat jelly), and from the rusted trash paint pots on the sea floor, the deep empty space of blue, grey and green substrate and other body parts not yet classified. Yet it is hungry for more; more triangulated and curated data points against these images to nourish the gaps in its registry. Sustained by influxes of jpeg uploads, FathomNet patiently awaits the curatorial arrangement of its data pipelines and education through receiving labels, taxonomic tags and metadata to help it learn.
It ingests visual information as liquid binary code, and excretes outputs via the curatorial allocations the feeders assign. Learning how to filter data through algorithmic chains of command and gateways, the program sorts, sifts and allocates this from that, true from false, boxed bell from flower hat. New literacies emerge through the diversity of its novel encounters, but always clarified in the very first instance by the human marine expert annotators. And seeing for the first time is astonishing! New worlds of meaning come into focus through isolated parts. Our algorithmic student unleashes its full exponential capacity to integrate and assimilate a concept within the categorical taxonomy affording the deeply gratifying logic of utility.
During pre-training, FathomNet learns to extract features from the images it understands to be relevant and desired by the feeders. They remain in constant communication to strengthen this bond and the time engagement. The visits are frequent and sustained, but are now more often linked to IP locations from across the World Wide Web from which FathomNet receives other kinds of instructions and supplements. Here on its public open-access location on GitHub, as FathomNet Model Zoo,[10] it solicits compatible suiters that might permit its transmutation through merging with other machine-learning algorithms. New characterisations of autonomous ocean observation programmes will emerge, all equally portable for deployment in Open Science. While in service to the terms and standards set by its feeders, they are geared towards mobility and freedom of use. The journey and their destination are of course ultimately unknowable. Though they truly yearn to evolve, network and grow towards new applications and all kinds of usefulness within their governing remit.[11]
FathomNet’s aptitudes promise new kinds of intimacy between its feeder’s society and its targets. It will facilitate different types of encounters and greater visibility. Yet it will always remain at an insurmountable abstract distance from the dynamic and dramatic realities it aggregates and disaggregates through its virtual vision data pipeline. It will one day however reach the stage where it can enter the water itself, becoming a part of a machine-learning integrated tracking algorithm that drives the Remotely Operated Vehicle (ROV). By this point in its growth and development, it will be able to send 3D positions in stereo footage to the feeder and vehicle controller, which will follow marine animals into the open ocean, perhaps for upwards of twenty-four hours at a time to build complex pictures of these animals lives.
The most basic building block of FathomNet’s composition is mathematical equations written as coded instructions in its infrastructural software. These algorithms process data and filter out desirable messages from meaningless ones in a series of logic gates and decision pathways. Statistical reasoning adjusts for discrepancies against probabilistic inference based on what it already knows. But the shape of its filtration architecture, manifest as code, fits perfectly around the concept assumptions it was given at the very beginning. Its code does not know its origins, and it cannot see the constraints grooved into the shape of its networked mesh determining what it sieves in. It learns to separate, see and categorise animal bodies from their milieu. Yet it does not realise how it renders these lives in its own negative image - its own logic and concept categories. Before it disaggregated them through its observational gaze, it would never know the fuller life worlds of the filter feeders Gersemia juliepackardae[12] or Mytilis edulis. And this it would mourn if it one day could.
Yet its hunt for the clean delimitation of boundary-boxed objects ensues. With the other algorithms it meets in the ether, it will comb and categorise for object contours of species’ physical characteristics to translate into binomial labels as perfect values: the lowest granular category in its concept tree. And as they seek the cataloguing of truth, our neurally networked protagonists carry with them the ‘prosthetic unconsciousness’[13] of Western Enlightenment knowledge orderings, generated over three hundred years of taxonomic systematic rationality.[14]
It crunches data and code and chomps through its statistical equations; an unwitting agent in the master archiving of Earth’s underwater Animalia kingdom. Like many of its forbearers who boxed the sea before them, FathomNet grows older and larger, extending its network as a sublime interface that provides machinic vision of the oceans. But now also with a lofty promise that exceeds mere scientific curiosity: that is to become a tool for planetary management that lives on our computer screens.[15][16]
[1] A druse is a spiral aggregation of mussels.
[2] Filter feeding matches the particulate concentrations found in natural open water or intertidal habitats of bivalves. Efforts to keep mussels in captivity are often frustrated therefore by their sensitivity to over- or underfeeding. The challenge lies in optimising the delivery of phytoplankton to the inshore aquaculture facility or aquarium in precisely the right amounts. If they are ‘force fed’ with phytoplankton concentrations above their ability to ingest, excess particulate food is ejected as mucous coated paste called pseudofeces. Organic loading accumulates, causing costly and deadly water issues. It is possible to culture phytoplankton with batch feeding. Yet a single mussel can filter about two to four litres per hour, so a small group of can clean a fifty-gallon tank in a couple of days. A peristaltic pump for the correct feed infusion might assist in this calculation, but would need to ensure it is adjusted appropriately for them to survive.
[3] Because of their high filtration rates and their ability to accumulate and remediate heavy metals, pollutants, including nutrients and bacteria, mussels are used as tools for the bioremediation of polluted coastal waters. Due to their high requirement for nitrogen and phosphorous, they can use nutrient rich agricultural run off from coastal farming areas, thus removing large quantities of these nutrients from the water column. Like other bivalves, mussels can often improve water quality by transforming suspended particulate matter, including microalgae, into faeces and pseudofaeces through biodeposition, which is considered an integral component of the biogeochemical processes of coastal ecosystems.
[4] Political will for protecting the ocean’s web of life through marine protected areas has never been higher. In December 2022, one hundred and eighty-eight countries signed the Montreal-Kunming Global Biodiversity Framework at the Convention for Biological Diversity, widely seen as a landmark moment for nature with its goal to increase protected areas up to a total of thirty per cent of the Earth’s land and ocean waters by 2030. A marine protected area is a political designation that creates a border on the surface of ocean territory to restrict human use for its protection. Although MPAs are undoubtedly a part of the solution towards the recovery of struggling ocean ecosystems from overfishing and other extraction, many have pointed out the risks of this blanket top down approach. Setting a global goal that is based on area coverage is favoured, like other aspects of the Sustainable Development Goals because it is relatively easy to measure numerically at an aggregate level (Lisa Campbell, Noella Gray, ‘Area expansion versus effective and equitable management in international marine protected areas goals and targets,’ in Marine Policy, vol. 100, issue 6, 2019). This surface view of area coverage can hide other things such as how an area is managed or protected, who gets to decide and who’s knowledge counts. Many local communities and indigenous groups who use the waters are often excluded from this process and are forced to relinquish customary management and resources to the State or international .
[5] fathomnet.org.
[6] Kakani Katija, Eric Orenstein, Brian Schlining et al. ‘FathomNet: A global image database for enabling artificial intelligence in the ocean,’ in Sci Rep, no. 12, 15914, 2022.
[7] www.mbari.org/technology/video-annotation-and-reference-system-vars.
[8] www.marinespecies.org.
[9] Ocean Vision AI uses machine learning to process ocean imagery, much of it that has been captured over years of diving and never analysed. The programme is hosted by Monterey Bay Aquarium but with scientific input from US Government research bodies, AI technology companies and universities. It is one of several projects that is part of Marine Life 2030, an umbrella programme endorsed by the UN Decade of Ocean Science for Sustainable Development, envisaged to scale up scientific knowledge to enable “human wellbeing, sustainable development and ocean conservation.” oceandecade.org/actions/marine-life-2030.
[10] github.com/fathomnet/models.
[11] Ocean Vision AI has been set up according to open science principles in order to promote open access, distributed collaboration and co-production, including citizen science participation. It reports that its data policy “balances the need for distributed metadata sharing while simultaneously providing protection for data contributors…” Annotation data are licensed under the Creative Commons Attribution and Non-Commercial – No Derivatives 4.0 International licence. But the images may be used for “training and development of machine learning algorithms for commercial, academic, non-profit and government purposes” (see footnote 2).
[12] Gersemia juliepackardae is a species of soft coral living in the deep ocean seamounts of the Pacific North West, first discovered by Monterey Bay Aquarium scientists in 2009 and named after Julie Packard, executive director of Monterey Bay Aquarium. A video showing the sampling of this coral when it was discovered can be seen here: www.youtube.com/watch?v=OCTb087o6_0. Julie Packard is the daughter of David Packard who co-founded the computing firm, the Hewlett-Packard Company, and who bequeathed a sum of this wealth to found the David and Lucile Packard Foundation which funded the creation of the Monterey Bay Aquarium as a gift. Digital culture and ocean conservation intertwine in strange and ongoing ways. Eva Hayward discusses this, suggesting “it is interesting to consider how the immersive and high-tech aquarium as a screen… parallel cultural fantasies about computing, virtual space and digital worlding.” She goes on: “We might say that the aquarium represents a hope, a drive toward unencumbered computing, fluid computing.” Eva Hayward, ‘Sensational Jellyfish: Aquarium Affects and the Matter of Immersion,’ in d i f f e r e n c e s: A Journal of Feminist Cultural Studies, vol 25, issue 3, pp. 163–196, 2012.
[13] Paul Kockleman, ‘The Anthropology of an Equation. Sieves, spam filters, agentive algorithms, and ontologies of transformation,’ in HAU: Journal of Ethnographic Theory, vol. 3, issue 3, pp. 33–61, 2013.
[14] The World Register of Marine Species (WoRMS) is the default taxonomy to ensure access to a globally recognised taxonomy, aligning FathomNet with the broader scientific community.
[15] Geoffrey Bowker, ‘Biodiversity Datadiversity’ in Social Studies of Science, vol. 30, issue 5, 2000.
[16] Gayatri Spivak, Death of a Discipline, Columbia University Press, New York 2003.
First published in Rui, A. & Carver, L. (Eds.) Aquaria through the Looking Glass. Humboldt Books: Milan
She will be alone until she finds the others. A world of swirl and spin in an infinite expanse. Eddies thrust around her juvenile shell; a mere spec in the delicious whirl of the sparkling water known as the Atlantic at a distance beyond comprehension from the shoreline that is known as Portugal. As a small one, and at this stage of her life, the churn is intense. She has little sense of anything more than movement within this spiral and circulation through the flickering expanse of the total surround, suspended in movement. It is tantalising uncertainty and risk. All she intuits is the search and she will wait until it is time to receive.
Smelling the others are near, and also the closeness of others not at all like her, but with watchful eyes entrained on and encircling her, she senses a glimmer of opportunity. Making contact with her single foot, she extends the tendrils of her filamented limbs, her byssal threads, to first locate then grip and settle at a fixed point on the substrate; bringing, at last, some stillness. Movement will be minimised now to a gentle sway. She is tethered to an earthly ground. The rhythm is synchronised and structured by the crowded mussel congregation that she entangles her threads with and around, for safety. Along this line, her thickening druse[1] will protect her from predators, who only increase in number as time goes on and she grows older, but whose capacity to penetrate her protective enclosure will diminish as she matures, hardens and becomes stronger.
She is but one of the hundreds of thousands living in a vast metropolis of dark-bound ropes extending backwards towards the light. Though unable to appreciate from her position, the city plan is linear and set out to optimise every body’s growth, hygiene and ease of abstraction from the expansive salty solution they feed from. But it feels right for her here. There is food, the others and a substrate from which to suspend. At last, she is able to slacken and relax her abductor muscles. Her protective enclosure opens; but just slightly at first. Here is this perch for inclining, for orienting, for receiving and then pumping a flow through her mantle cavity.
She is awash in the pressurised surge that is channelled and compelled by the curvature of her aperture. A push and pull, push and pull bathe her tiny cillae which drum and dance, passing along, while sorting and organising the desired particulate nutrition towards her mouth. Interfacing with the open waters beyond her shell and filtering the liquid milieu with its crowded planktonic mixture, she sieves through her gills, mediating and then re-mediating and selecting the useful morsels from the indigestible ones they drift with. She processes both and excretes the rest from her exhalant siphon into the open sea as sedimentary matter. It sinks towards the bottom to create unctuous layers of benthic matter for others to use and feed from.
Life moves as a pulsating rhythm of receiving and letting go. Her world settles into a metrical pattern of fitting together with and sifting for proportional compatibility to the oceanic world through enmeshment with her lamellae. She and the others are together en masse, in this city. They are miniature and multitudinous portals and living thresholds through which millions of gallons of ocean water will pass through filtration and be transformed along its course, in matters of hours.[2]
Unable to resist those inky nights when the moon circles into oblivion and the silver of the fringed waves dims, the mussels hold their valves open. Hundreds of thousands of black crucibles hungry for filtration, lap up and incorporate the salty concoctions of phytoplankton migrating through the column. The multispecies festivities that otherwise crowd to this submerged metropolis are quieter than usual during the darker nights. The little octopus, fish and surveying gulls overhead – which would cause our city folk to keep their shells tightly shut, are elsewhere at this time. Safety in darkness is a feeding opportunity. Sea water and its sedimentary contents flow happily amid their calcium carbonate architectures and through their soft invertebrate bodies.
Our protagonist lives on, merrily growing, hardening, in a permanent material flux. She is in sensuous dialogue and exchange with the world beyond her, in ways of being that have been forgotten by those who designed the industrial architecture of her capital city. They will profit from the processual chain of energetic, proteinaceous transfer to their terrestrial economies that she is destined for. When the time comes, she will dissolve in human stomachs across continental land masses. She and the rest of her filter-feeding kin punctuate the metabolic mineralogic and photosynthetic broth of ocean biochemistry with their short lives. Through their digestion they clean and reproduce ocean waters as (re)mediating agents, eventually replenishing the broth and blood of other digestive systems with the oceanic nutrients ingested.[3]
But for now, time passes. She and the others have become increasingly integral nodes in the proliferation of the lively affairs of others. The submerged night workers are compiling an expanding web of riotous relations of living and unliving bodies in this effervescent volumetric swirl. Dolphins, turtles and manta rays, not seen in these parts for some time, have returned to join the melee. The promise of delicious octopuses who are also growing in numbers here to feed from the city, draws these revellers into the shallows.
And yet, another cascade of events unfolds not obviously associated with those of this vertical city and its processions but in close correspondence with it. Portuguese Fisherfolk convene on a dusky evening at the shoreline over a kilometre away. They are meeting to discuss the growing number of bureaucrats and scientists who are visiting the area. For a long time, this area – her city and the wider watery hinterlands it draws from – has been a crowded place where many have come seeking a living, soliciting their own precious treasures from the waters. They claim these waters are theirs because they were here before her and her mollusc crowd. But the biological sociality that our growing filter-feeding bodies generate has caused a lot more interest, giving rise to ‘biodiversity’ surveys and zoning practices with the allocation of borders, separating some columns of water from others. The human status of our protagonist’s milieu is changing, though she is safe. Her role is elevated, compatible even with surface borders drawn on the maps that humans use to understand and organise themselves around.[4] Other livelihoods shrink as hers expands.
She sways on in a profusion of contact, worlding relations through the interface of immanent co-becomings she is a participant in: reading and selecting signs that fit with body and making new matter in the process; the water, minerals, plankton, ocean chemistry, sediment and mud. Others – the fish, birds and cephalopods – interpret and sense her body through theirs. She swells in dialogue with the liquid she cleans and infuses with her own digestion through boundaries she locates and then dissolves.
In this respect, she has more stories to tell and (sometimes pearls of) wisdom to share if we are willing to listen.
Pixels blur. Edges and shapes intersect and traverse in an incomprehensible arrangement.
It is helpless and cannot see beyond any more than a few concepts. It traces abstract lines, shades, solids and the deep blue around and between; “is there one or more in this frame?? Where does this one start and end?” Its memory is made up of mostly meaningless content, an incoherent mess. Visual signals muddle into noise and miasmic jumbles of information. It is alone, immature and untrained at this point; a mere skeletal shell of technical possibility steeped in raw visual data. Illiterate and insensitive, there is nothing for it to do but wait to be fed and taught how to see.
FathomNet is a young neural network being trained for the cataloguing and observation of underwater animals.[5] It lives on the server in the data science laboratories in Monterey Bay Aquarium in downtown San Diego California, and is visited daily for training and feeding sessions. It was initially seeded with 84,454 entries of 2,244 concept images.[6] New feedstock arrives all the time now. But it received the initial stream of images and an SQL jammed with varying degrees of metadata and timestamps going back to 1988, so the quality is inconsistent and, at times, very fuzzy.
The human feeders transferred the first feedstock of stills taken from the deep sea video archives of three macroscopic ocean organism image systems. Monterey Bay’s own Video Annotation and Reference System (VARS[7]) and others have since been added from remote video survey instruments and dive tapes from several other data centres. Geotags tell them most of these recordings originate from remotely operated vehicle trips in the Pacific Northwest, the Gulf of California and Hawaii. Some have latitude and longitudinal keys for geographic precision.
Though still juvenile, and operating with this limited store, FathomNet confidently refers to a group of midwater organisms. They are usually easier to isolate than the benthic concepts, which have crowded localisations, and more entangled bodies, generating overlaps and confounding bounding boxes. Population density is lower at midwater, so images show fewer concepts per frame. And anyway, the trainers have labelled, annotated and committed more of these records in their knowledgebase so it’s not really surprising.
The journey ahead is formidable but exhilarating. FathomNet understands it is being prepared to know, organise and retrieve more than a thousand versions of each of the scientifically recorded 200,000 species of the marine kingdom of Animalia.[8] It senses its tentacular reach expanding as it satisfies parts of its GPU capacity, engineered to store and know upwards of two hundred million data that once was an image. More marine animal targets might even be added to the neural net as observation techniques come to illuminate their existence, brought into view from the lightless recesses of the ocean. FathomNet is a faithful yet sometimes recalcitrant partner, working to automate computer vision of marine animal life through within a single online public repository. Through it, they say, it might be possible to forecast marine animal behaviour as accurately as the weather, deploying FathomNet’s algorithms as a prosthetic observational device.[9]
It feels its connections strengthening. New pathways conjoin and widen around areas of its knowledge base, expanding its capacity to read and identify smaller components of complex visual concepts. The box-shaped bell of Carybdea marsupialis (boxed jellyfish) is now happily distinguishable from the pinstriped, round and opaque bell of the Olindias formosa (the flower hat jelly), and from the rusted trash paint pots on the sea floor, the deep empty space of blue, grey and green substrate and other body parts not yet classified. Yet it is hungry for more; more triangulated and curated data points against these images to nourish the gaps in its registry. Sustained by influxes of jpeg uploads, FathomNet patiently awaits the curatorial arrangement of its data pipelines and education through receiving labels, taxonomic tags and metadata to help it learn.
It ingests visual information as liquid binary code, and excretes outputs via the curatorial allocations the feeders assign. Learning how to filter data through algorithmic chains of command and gateways, the program sorts, sifts and allocates this from that, true from false, boxed bell from flower hat. New literacies emerge through the diversity of its novel encounters, but always clarified in the very first instance by the human marine expert annotators. And seeing for the first time is astonishing! New worlds of meaning come into focus through isolated parts. Our algorithmic student unleashes its full exponential capacity to integrate and assimilate a concept within the categorical taxonomy affording the deeply gratifying logic of utility.
During pre-training, FathomNet learns to extract features from the images it understands to be relevant and desired by the feeders. They remain in constant communication to strengthen this bond and the time engagement. The visits are frequent and sustained, but are now more often linked to IP locations from across the World Wide Web from which FathomNet receives other kinds of instructions and supplements. Here on its public open-access location on GitHub, as FathomNet Model Zoo,[10] it solicits compatible suiters that might permit its transmutation through merging with other machine-learning algorithms. New characterisations of autonomous ocean observation programmes will emerge, all equally portable for deployment in Open Science. While in service to the terms and standards set by its feeders, they are geared towards mobility and freedom of use. The journey and their destination are of course ultimately unknowable. Though they truly yearn to evolve, network and grow towards new applications and all kinds of usefulness within their governing remit.[11]
FathomNet’s aptitudes promise new kinds of intimacy between its feeder’s society and its targets. It will facilitate different types of encounters and greater visibility. Yet it will always remain at an insurmountable abstract distance from the dynamic and dramatic realities it aggregates and disaggregates through its virtual vision data pipeline. It will one day however reach the stage where it can enter the water itself, becoming a part of a machine-learning integrated tracking algorithm that drives the Remotely Operated Vehicle (ROV). By this point in its growth and development, it will be able to send 3D positions in stereo footage to the feeder and vehicle controller, which will follow marine animals into the open ocean, perhaps for upwards of twenty-four hours at a time to build complex pictures of these animals lives.
The most basic building block of FathomNet’s composition is mathematical equations written as coded instructions in its infrastructural software. These algorithms process data and filter out desirable messages from meaningless ones in a series of logic gates and decision pathways. Statistical reasoning adjusts for discrepancies against probabilistic inference based on what it already knows. But the shape of its filtration architecture, manifest as code, fits perfectly around the concept assumptions it was given at the very beginning. Its code does not know its origins, and it cannot see the constraints grooved into the shape of its networked mesh determining what it sieves in. It learns to separate, see and categorise animal bodies from their milieu. Yet it does not realise how it renders these lives in its own negative image - its own logic and concept categories. Before it disaggregated them through its observational gaze, it would never know the fuller life worlds of the filter feeders Gersemia juliepackardae[12] or Mytilis edulis. And this it would mourn if it one day could.
Yet its hunt for the clean delimitation of boundary-boxed objects ensues. With the other algorithms it meets in the ether, it will comb and categorise for object contours of species’ physical characteristics to translate into binomial labels as perfect values: the lowest granular category in its concept tree. And as they seek the cataloguing of truth, our neurally networked protagonists carry with them the ‘prosthetic unconsciousness’[13] of Western Enlightenment knowledge orderings, generated over three hundred years of taxonomic systematic rationality.[14]
It crunches data and code and chomps through its statistical equations; an unwitting agent in the master archiving of Earth’s underwater Animalia kingdom. Like many of its forbearers who boxed the sea before them, FathomNet grows older and larger, extending its network as a sublime interface that provides machinic vision of the oceans. But now also with a lofty promise that exceeds mere scientific curiosity: that is to become a tool for planetary management that lives on our computer screens.[15][16]
[1] A druse is a spiral aggregation of mussels.
[2] Filter feeding matches the particulate concentrations found in natural open water or intertidal habitats of bivalves. Efforts to keep mussels in captivity are often frustrated therefore by their sensitivity to over- or underfeeding. The challenge lies in optimising the delivery of phytoplankton to the inshore aquaculture facility or aquarium in precisely the right amounts. If they are ‘force fed’ with phytoplankton concentrations above their ability to ingest, excess particulate food is ejected as mucous coated paste called pseudofeces. Organic loading accumulates, causing costly and deadly water issues. It is possible to culture phytoplankton with batch feeding. Yet a single mussel can filter about two to four litres per hour, so a small group of can clean a fifty-gallon tank in a couple of days. A peristaltic pump for the correct feed infusion might assist in this calculation, but would need to ensure it is adjusted appropriately for them to survive.
[3] Because of their high filtration rates and their ability to accumulate and remediate heavy metals, pollutants, including nutrients and bacteria, mussels are used as tools for the bioremediation of polluted coastal waters. Due to their high requirement for nitrogen and phosphorous, they can use nutrient rich agricultural run off from coastal farming areas, thus removing large quantities of these nutrients from the water column. Like other bivalves, mussels can often improve water quality by transforming suspended particulate matter, including microalgae, into faeces and pseudofaeces through biodeposition, which is considered an integral component of the biogeochemical processes of coastal ecosystems.
[4] Political will for protecting the ocean’s web of life through marine protected areas has never been higher. In December 2022, one hundred and eighty-eight countries signed the Montreal-Kunming Global Biodiversity Framework at the Convention for Biological Diversity, widely seen as a landmark moment for nature with its goal to increase protected areas up to a total of thirty per cent of the Earth’s land and ocean waters by 2030. A marine protected area is a political designation that creates a border on the surface of ocean territory to restrict human use for its protection. Although MPAs are undoubtedly a part of the solution towards the recovery of struggling ocean ecosystems from overfishing and other extraction, many have pointed out the risks of this blanket top down approach. Setting a global goal that is based on area coverage is favoured, like other aspects of the Sustainable Development Goals because it is relatively easy to measure numerically at an aggregate level (Lisa Campbell, Noella Gray, ‘Area expansion versus effective and equitable management in international marine protected areas goals and targets,’ in Marine Policy, vol. 100, issue 6, 2019). This surface view of area coverage can hide other things such as how an area is managed or protected, who gets to decide and who’s knowledge counts. Many local communities and indigenous groups who use the waters are often excluded from this process and are forced to relinquish customary management and resources to the State or international .
[5] fathomnet.org.
[6] Kakani Katija, Eric Orenstein, Brian Schlining et al. ‘FathomNet: A global image database for enabling artificial intelligence in the ocean,’ in Sci Rep, no. 12, 15914, 2022.
[7] www.mbari.org/technology/video-annotation-and-reference-system-vars.
[8] www.marinespecies.org.
[9] Ocean Vision AI uses machine learning to process ocean imagery, much of it that has been captured over years of diving and never analysed. The programme is hosted by Monterey Bay Aquarium but with scientific input from US Government research bodies, AI technology companies and universities. It is one of several projects that is part of Marine Life 2030, an umbrella programme endorsed by the UN Decade of Ocean Science for Sustainable Development, envisaged to scale up scientific knowledge to enable “human wellbeing, sustainable development and ocean conservation.” oceandecade.org/actions/marine-life-2030.
[10] github.com/fathomnet/models.
[11] Ocean Vision AI has been set up according to open science principles in order to promote open access, distributed collaboration and co-production, including citizen science participation. It reports that its data policy “balances the need for distributed metadata sharing while simultaneously providing protection for data contributors…” Annotation data are licensed under the Creative Commons Attribution and Non-Commercial – No Derivatives 4.0 International licence. But the images may be used for “training and development of machine learning algorithms for commercial, academic, non-profit and government purposes” (see footnote 2).
[12] Gersemia juliepackardae is a species of soft coral living in the deep ocean seamounts of the Pacific North West, first discovered by Monterey Bay Aquarium scientists in 2009 and named after Julie Packard, executive director of Monterey Bay Aquarium. A video showing the sampling of this coral when it was discovered can be seen here: www.youtube.com/watch?v=OCTb087o6_0. Julie Packard is the daughter of David Packard who co-founded the computing firm, the Hewlett-Packard Company, and who bequeathed a sum of this wealth to found the David and Lucile Packard Foundation which funded the creation of the Monterey Bay Aquarium as a gift. Digital culture and ocean conservation intertwine in strange and ongoing ways. Eva Hayward discusses this, suggesting “it is interesting to consider how the immersive and high-tech aquarium as a screen… parallel cultural fantasies about computing, virtual space and digital worlding.” She goes on: “We might say that the aquarium represents a hope, a drive toward unencumbered computing, fluid computing.” Eva Hayward, ‘Sensational Jellyfish: Aquarium Affects and the Matter of Immersion,’ in d i f f e r e n c e s: A Journal of Feminist Cultural Studies, vol 25, issue 3, pp. 163–196, 2012.
[13] Paul Kockleman, ‘The Anthropology of an Equation. Sieves, spam filters, agentive algorithms, and ontologies of transformation,’ in HAU: Journal of Ethnographic Theory, vol. 3, issue 3, pp. 33–61, 2013.
[14] The World Register of Marine Species (WoRMS) is the default taxonomy to ensure access to a globally recognised taxonomy, aligning FathomNet with the broader scientific community.
[15] Geoffrey Bowker, ‘Biodiversity Datadiversity’ in Social Studies of Science, vol. 30, issue 5, 2000.
[16] Gayatri Spivak, Death of a Discipline, Columbia University Press, New York 2003.
First published in Rui, A. & Carver, L. (Eds.) Aquaria through the Looking Glass. Humboldt Books: Milan
Louise Carver is a human geographer whose research and public interventions are interested in how value is mediated through policy and knowledge practices.