{
"nbformat": 4,
"nbformat_minor": 4,
"metadata": {
"title": "A crowd of BashTheBug volunteers reproducibly and accurately measure the minimum inhibitory concentrations of 13 antitubercular drugs from photographs of 96-well broth microdilution plates",
"authors": [
{
"givenNames": [
"Philip",
"W"
],
"familyNames": [
"Fowler"
],
"type": "Person",
"emails": [
"philip.fowler@ndm.ox.ac.uk"
],
"affiliations": [
{
"name": "Oxford",
"address": {
"addressCountry": "United Kingdom",
"type": "PostalAddress"
},
"type": "Organization"
}
],
"name": "Philip W Fowler"
},
{
"givenNames": [
"Carla"
],
"familyNames": [
"Wright"
],
"type": "Person",
"affiliations": [
{
"name": "Oxford",
"address": {
"addressCountry": "United Kingdom",
"type": "PostalAddress"
},
"type": "Organization"
}
],
"name": "Carla Wright"
},
{
"givenNames": [
"Helen"
],
"familyNames": [
"Spiers"
],
"type": "Person",
"affiliations": [
{
"name": "Oxford",
"address": {
"addressCountry": "United Kingdom",
"type": "PostalAddress"
},
"type": "Organization"
},
{
"name": "London",
"address": {
"addressCountry": "United Kingdom",
"type": "PostalAddress"
},
"type": "Organization"
}
],
"name": "Helen Spiers"
},
{
"givenNames": [
"Tingting"
],
"familyNames": [
"Zhu"
],
"type": "Person",
"affiliations": [
{
"name": "Oxford",
"address": {
"addressCountry": "United Kingdom",
"type": "PostalAddress"
},
"type": "Organization"
}
],
"name": "Tingting Zhu"
},
{
"givenNames": [
"Elisabeth",
"ML"
],
"familyNames": [
"Baeten"
],
"type": "Person",
"affiliations": [
{
"name": "Oxford",
"address": {
"addressCountry": "United Kingdom",
"type": "PostalAddress"
},
"type": "Organization"
}
],
"name": "Elisabeth ML Baeten"
},
{
"givenNames": [
"Sarah",
"W"
],
"familyNames": [
"Hoosdally"
],
"type": "Person",
"affiliations": [
{
"name": "Oxford",
"address": {
"addressCountry": "United Kingdom",
"type": "PostalAddress"
},
"type": "Organization"
}
],
"name": "Sarah W Hoosdally"
},
{
"givenNames": [
"Ana",
"L"
],
"familyNames": [
"Gibertoni",
"Cruz"
],
"type": "Person",
"affiliations": [
{
"name": "Oxford",
"address": {
"addressCountry": "United Kingdom",
"type": "PostalAddress"
},
"type": "Organization"
}
],
"name": "Ana L Gibertoni Cruz"
},
{
"givenNames": [
"Aysha"
],
"familyNames": [
"Roohi"
],
"type": "Person",
"affiliations": [
{
"name": "Oxford",
"address": {
"addressCountry": "United Kingdom",
"type": "PostalAddress"
},
"type": "Organization"
}
],
"name": "Aysha Roohi"
},
{
"givenNames": [
"Samaneh"
],
"familyNames": [
"Kouchaki"
],
"type": "Person",
"affiliations": [
{
"name": "Oxford",
"address": {
"addressCountry": "United Kingdom",
"type": "PostalAddress"
},
"type": "Organization"
}
],
"name": "Samaneh Kouchaki"
},
{
"givenNames": [
"Timothy",
"M"
],
"familyNames": [
"Walker"
],
"type": "Person",
"affiliations": [
{
"name": "Oxford",
"address": {
"addressCountry": "United Kingdom",
"type": "PostalAddress"
},
"type": "Organization"
}
],
"name": "Timothy M Walker"
},
{
"givenNames": [
"Timothy",
"EA"
],
"familyNames": [
"Peto"
],
"type": "Person",
"affiliations": [
{
"name": "Oxford",
"address": {
"addressCountry": "United Kingdom",
"type": "PostalAddress"
},
"type": "Organization"
}
],
"name": "Timothy EA Peto"
},
{
"givenNames": [
"Grant"
],
"familyNames": [
"Miller"
],
"type": "Person",
"affiliations": [
{
"name": "Oxford",
"address": {
"addressCountry": "United Kingdom",
"type": "PostalAddress"
},
"type": "Organization"
}
],
"name": "Grant Miller"
},
{
"givenNames": [
"Chris"
],
"familyNames": [
"Lintott"
],
"type": "Person",
"affiliations": [
{
"name": "Oxford",
"address": {
"addressCountry": "United Kingdom",
"type": "PostalAddress"
},
"type": "Organization"
}
],
"name": "Chris Lintott"
},
{
"givenNames": [
"David"
],
"familyNames": [
"Clifton"
],
"type": "Person",
"affiliations": [
{
"name": "Oxford",
"address": {
"addressCountry": "United Kingdom",
"type": "PostalAddress"
},
"type": "Organization"
}
],
"name": "David Clifton"
},
{
"givenNames": [
"Derrick",
"W"
],
"familyNames": [
"Crook"
],
"type": "Person",
"affiliations": [
{
"name": "Oxford",
"address": {
"addressCountry": "United Kingdom",
"type": "PostalAddress"
},
"type": "Organization"
}
],
"name": "Derrick W Crook"
},
{
"givenNames": [
"A",
"Sarah"
],
"familyNames": [
"Walker"
],
"type": "Person",
"affiliations": [
{
"name": "Oxford",
"address": {
"addressCountry": "United Kingdom",
"type": "PostalAddress"
},
"type": "Organization"
}
],
"name": "A Sarah Walker"
},
{
"name": "The Zooniverse Volunteer Community",
"type": "Organization"
},
{
"name": "The CRyPTIC Consortium",
"members": [
{
"givenNames": [
"Ivan"
],
"familyNames": [
"Barilar"
],
"type": "Person",
"affiliations": [
{
"name": "Research Center Borstel",
"address": {
"addressCountry": "Germany",
"addressLocality": "Borstel",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Simone"
],
"familyNames": [
"Battaglia"
],
"type": "Person",
"affiliations": [
{
"name": "IRCCS San Raffaele Scientific Institute",
"address": {
"addressCountry": "Italy",
"addressLocality": "Milan",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Emanuele"
],
"familyNames": [
"Borroni"
],
"type": "Person",
"affiliations": [
{
"name": "IRCCS San Raffaele Scientific Institute",
"address": {
"addressCountry": "Italy",
"addressLocality": "Milan",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Angela",
"Pires"
],
"familyNames": [
"Brandao"
],
"type": "Person",
"affiliations": [
{
"name": "Oswaldo Cruz Foundation",
"address": {
"addressCountry": "Brazil",
"addressLocality": "Rio de Janeiro",
"type": "PostalAddress"
},
"type": "Organization"
},
{
"name": "Institute Adolfo Lutz",
"address": {
"addressCountry": "Brazil",
"addressLocality": "Sao Paulo",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Alice"
],
"familyNames": [
"Brankin"
],
"type": "Person",
"affiliations": [
{
"name": "University of Oxford",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Oxford",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Andrea",
"Maurizio"
],
"familyNames": [
"Cabibbe"
],
"type": "Person",
"affiliations": [
{
"name": "IRCCS San Raffaele Scientific Institute",
"address": {
"addressCountry": "Italy",
"addressLocality": "Milan",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Joshua"
],
"familyNames": [
"Carter"
],
"type": "Person",
"affiliations": [
{
"name": "Stanford University School of Medicine",
"address": {
"addressCountry": "United States",
"addressLocality": "Stanford",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Daniela",
"Maria"
],
"familyNames": [
"Cirillo"
],
"type": "Person",
"affiliations": [
{
"name": "IRCCS San Raffaele Scientific Institute",
"address": {
"addressCountry": "Italy",
"addressLocality": "Milan",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Pauline"
],
"familyNames": [
"Claxton"
],
"type": "Person",
"affiliations": [
{
"name": "Scottish Mycobacteria Reference Laboratory",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Edinburgh",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"David",
"A"
],
"familyNames": [
"Clifton"
],
"type": "Person",
"affiliations": [
{
"name": "University of Oxford",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Oxford",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Ted"
],
"familyNames": [
"Cohen"
],
"type": "Person",
"affiliations": [
{
"name": "Yale School of Public Health",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Yale",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Jorge"
],
"familyNames": [
"Coronel"
],
"type": "Person",
"affiliations": [
{
"name": "Universidad Peruana Cayetano Heredia",
"address": {
"addressCountry": "Peru",
"addressLocality": "Lima",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Derrick",
"W"
],
"familyNames": [
"Crook"
],
"type": "Person",
"affiliations": [
{
"name": "University of Oxford",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Oxford",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Sarah",
"G"
],
"familyNames": [
"Earle"
],
"type": "Person",
"affiliations": [
{
"name": "University of Oxford",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Oxford",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Vincent"
],
"familyNames": [
"Escuyer"
],
"type": "Person",
"affiliations": [
{
"name": "Wadsworth Center, New York State Department of Health",
"address": {
"addressCountry": "United States",
"addressLocality": "Albany",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Lucilaine"
],
"familyNames": [
"Ferrazoli"
],
"type": "Person",
"affiliations": [
{
"name": "Institute Adolfo Lutz",
"address": {
"addressCountry": "Brazil",
"addressLocality": "Sao Paulo",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Philip",
"W"
],
"familyNames": [
"Fowler"
],
"type": "Person",
"affiliations": [
{
"name": "University of Oxford",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Oxford",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"George",
"F"
],
"familyNames": [
"Gao"
],
"type": "Person",
"affiliations": [
{
"name": "Chinese Center for Disease Control and Prevention",
"address": {
"addressCountry": "China",
"addressLocality": "Beijing",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Jennifer"
],
"familyNames": [
"Gardy"
],
"type": "Person",
"affiliations": [
{
"name": "Seattle",
"address": {
"addressCountry": "United States",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Saheer"
],
"familyNames": [
"Gharbia"
],
"type": "Person",
"affiliations": [
{
"name": "UK Health Security Agency",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "London",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Kelen",
"Teixeira"
],
"familyNames": [
"Ghisi"
],
"type": "Person",
"affiliations": [
{
"name": "Institute Adolfo Lutz",
"address": {
"addressCountry": "Brazil",
"addressLocality": "Sao Paulo",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Arash"
],
"familyNames": [
"Ghodousi"
],
"type": "Person",
"affiliations": [
{
"name": "IRCCS San Raffaele Scientific Institute",
"address": {
"addressCountry": "Italy",
"addressLocality": "Milan",
"type": "PostalAddress"
},
"type": "Organization"
},
{
"name": "Vita-Salute San Raffaele University",
"address": {
"addressCountry": "Italy",
"addressLocality": "Milan",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Ana",
"Luıza"
],
"familyNames": [
"Gibertoni",
"Cruz"
],
"type": "Person",
"affiliations": [
{
"name": "University of Oxford",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Oxford",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Clara"
],
"familyNames": [
"Grazian"
],
"type": "Person",
"affiliations": [
{
"name": "University of New South Wales",
"address": {
"addressCountry": "Australia",
"addressLocality": "Sydney",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Jennifer",
"L"
],
"familyNames": [
"Guthrie"
],
"type": "Person",
"affiliations": [
{
"name": "The University of British Columbia",
"address": {
"addressCountry": "Canada",
"addressLocality": "Vancouver",
"type": "PostalAddress"
},
"type": "Organization"
},
{
"name": "Public Health Ontario",
"address": {
"addressCountry": "Canada",
"addressLocality": "Toronto",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Wencong"
],
"familyNames": [
"He"
],
"type": "Person",
"affiliations": [
{
"name": "Chinese Center for Disease Control and Prevention",
"address": {
"addressCountry": "China",
"addressLocality": "Beijing",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Harald"
],
"familyNames": [
"Hoffmann"
],
"type": "Person",
"affiliations": [
{
"name": "SYNLAB Gauting",
"address": {
"addressCountry": "Germany",
"addressLocality": "Munich",
"type": "PostalAddress"
},
"type": "Organization"
},
{
"name": "Institute of Microbiology and Laboratory Medicine,WHO-SRL Gauting",
"address": {
"addressCountry": "Germany",
"addressLocality": "IMLred",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Sarah",
"J"
],
"familyNames": [
"Hoosdally"
],
"type": "Person",
"affiliations": [
{
"name": "University of Oxford",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Oxford",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Martin"
],
"familyNames": [
"Hunt"
],
"type": "Person",
"affiliations": [
{
"name": "University of Oxford",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Oxford",
"type": "PostalAddress"
},
"type": "Organization"
},
{
"name": "EMBL-EBI",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Hinxton",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Zamin"
],
"familyNames": [
"Iqbal"
],
"type": "Person",
"affiliations": [
{
"name": "EMBL-EBI",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Hinxton",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Nazir",
"Ahmed"
],
"familyNames": [
"Ismail"
],
"type": "Person",
"affiliations": [
{
"name": "National Institute for Communicable Diseases",
"address": {
"addressCountry": "South Africa",
"addressLocality": "Johannesburg",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Lisa"
],
"familyNames": [
"Jarrett"
],
"type": "Person",
"affiliations": [
{
"name": "UK Health Security Agency",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Birmingham",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Lavania"
],
"familyNames": [
"Joseph"
],
"type": "Person",
"affiliations": [
{
"name": "National Institute for Communicable Diseases",
"address": {
"addressCountry": "South Africa",
"addressLocality": "Johannesburg",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Ruwen"
],
"familyNames": [
"Jou"
],
"type": "Person",
"affiliations": [
{
"name": "Taiwan Centers for Disease Control",
"address": {
"addressCountry": "Taiwan",
"addressLocality": "Taipei",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Priti"
],
"familyNames": [
"Kambli"
],
"type": "Person",
"affiliations": [
{
"name": "Hinduja Hospital",
"address": {
"addressCountry": "India",
"addressLocality": "Mumbai",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Jeff"
],
"familyNames": [
"Knaggs"
],
"type": "Person",
"affiliations": [
{
"name": "University of Oxford",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Oxford",
"type": "PostalAddress"
},
"type": "Organization"
},
{
"name": "EMBL-EBI",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Hinxton",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Anastasia"
],
"familyNames": [
"Koch"
],
"type": "Person",
"affiliations": [
{
"name": "Cape Town",
"address": {
"addressCountry": "South Africa",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Donna"
],
"familyNames": [
"Kohlerschmidt"
],
"type": "Person",
"affiliations": [
{
"name": "Wadsworth Center, New York State Department of Health",
"address": {
"addressCountry": "United States",
"addressLocality": "Albany",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Samaneh"
],
"familyNames": [
"Kouchaki"
],
"type": "Person",
"affiliations": [
{
"name": "University of Oxford",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Oxford",
"type": "PostalAddress"
},
"type": "Organization"
},
{
"name": "University of Surrey",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Guildford",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Alexander",
"S"
],
"familyNames": [
"Lachapelle"
],
"type": "Person",
"affiliations": [
{
"name": "University of Oxford",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Oxford",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Ajit"
],
"familyNames": [
"Lalvani"
],
"type": "Person",
"affiliations": [
{
"name": "Imperial College",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "London",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Simon",
"Grandjean"
],
"familyNames": [
"Lapierre"
],
"type": "Person",
"affiliations": [
{
"name": "Universite de Montreal",
"address": {
"addressCountry": "United States",
"addressLocality": "Canada",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Ian",
"F"
],
"familyNames": [
"Laurenson"
],
"type": "Person",
"affiliations": [
{
"name": "Scottish Mycobacteria Reference Laboratory",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Edinburgh",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Brice"
],
"familyNames": [
"Letcher"
],
"type": "Person",
"affiliations": [
{
"name": "EMBL-EBI",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Hinxton",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Wan-Hsuan"
],
"familyNames": [
"Lin"
],
"type": "Person",
"affiliations": [
{
"name": "Taiwan Centers for Disease Control",
"address": {
"addressCountry": "Taiwan",
"addressLocality": "Taipei",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Chunfa"
],
"familyNames": [
"Liu"
],
"type": "Person",
"affiliations": [
{
"name": "Chinese Center for Disease Control and Prevention",
"address": {
"addressCountry": "China",
"addressLocality": "Beijing",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Dongxin"
],
"familyNames": [
"Liu"
],
"type": "Person",
"affiliations": [
{
"name": "Chinese Center for Disease Control and Prevention",
"address": {
"addressCountry": "China",
"addressLocality": "Beijing",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Kerri",
"M"
],
"familyNames": [
"Malone"
],
"type": "Person",
"affiliations": [
{
"name": "EMBL-EBI",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Hinxton",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Ayan"
],
"familyNames": [
"Mandal"
],
"type": "Person",
"affiliations": [
{
"name": "The Foundation for Medical Research",
"address": {
"addressCountry": "India",
"addressLocality": "Mumbai",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Daniela"
],
"familyNames": [
"Matias"
],
"type": "Person",
"affiliations": [
{
"name": "UK Health Security Agency",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Birmingham",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Graeme"
],
"familyNames": [
"Meintjes"
],
"type": "Person",
"affiliations": [
{
"name": "Cape Town",
"address": {
"addressCountry": "South Africa",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Flavia"
],
"familyNames": [
"Freitas",
"Mendes"
],
"type": "Person",
"affiliations": [
{
"name": "Institute Adolfo Lutz",
"address": {
"addressCountry": "Brazil",
"addressLocality": "Sao Paulo",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Matthias"
],
"familyNames": [
"Merker"
],
"type": "Person",
"affiliations": [
{
"name": "Research Center Borstel",
"address": {
"addressCountry": "Germany",
"addressLocality": "Borstel",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Marina"
],
"familyNames": [
"Mihalic"
],
"type": "Person",
"affiliations": [
{
"name": "Institute of Microbiology and Laboratory Medicine,WHO-SRL Gauting",
"address": {
"addressCountry": "Germany",
"addressLocality": "IMLred",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"James"
],
"familyNames": [
"Millard"
],
"type": "Person",
"affiliations": [
{
"name": "Africa Health Research Institute",
"address": {
"addressCountry": "South Africa",
"addressLocality": "Durban",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Paolo"
],
"familyNames": [
"Miotto"
],
"type": "Person",
"affiliations": [
{
"name": "IRCCS San Raffaele Scientific Institute",
"address": {
"addressCountry": "Italy",
"addressLocality": "Milan",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Nerges"
],
"familyNames": [
"Mistry"
],
"type": "Person",
"affiliations": [
{
"name": "The Foundation for Medical Research",
"address": {
"addressCountry": "India",
"addressLocality": "Mumbai",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"David"
],
"familyNames": [
"Moore"
],
"type": "Person",
"affiliations": [
{
"name": "Universidad Peruana Cayetano Heredia",
"address": {
"addressCountry": "Peru",
"addressLocality": "Lima",
"type": "PostalAddress"
},
"type": "Organization"
},
{
"name": "London School of Hygiene and Tropical Medicine",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "London",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Viola"
],
"familyNames": [
"Dreyer"
],
"type": "Person",
"affiliations": [
{
"name": "Research Center Borstel",
"address": {
"addressCountry": "Germany",
"addressLocality": "Borstel",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Darren"
],
"familyNames": [
"Chetty"
],
"type": "Person",
"affiliations": [
{
"name": "Africa Health Research Institute",
"address": {
"addressCountry": "South Africa",
"addressLocality": "Durban",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Kimberlee",
"A"
],
"familyNames": [
"Musser"
],
"type": "Person",
"affiliations": [
{
"name": "Wadsworth Center, New York State Department of Health",
"address": {
"addressCountry": "United States",
"addressLocality": "Albany",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Dumisani"
],
"familyNames": [
"Ngcamu"
],
"type": "Person",
"affiliations": [
{
"name": "National Institute for Communicable Diseases",
"address": {
"addressCountry": "South Africa",
"addressLocality": "Johannesburg",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Hoang",
"Ngoc"
],
"familyNames": [
"Nhung"
],
"type": "Person",
"affiliations": [
{
"name": "Oxford University Clinical Research Unit",
"address": {
"addressCountry": "Viet Nam",
"addressLocality": "Ho Chi Minh City",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Louis"
],
"familyNames": [
"Grandjean"
],
"type": "Person",
"affiliations": [
{
"name": "University College London",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "London",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Kayzad",
"Soli"
],
"familyNames": [
"Nilgiriwala"
],
"type": "Person",
"affiliations": [
{
"name": "The Foundation for Medical Research",
"address": {
"addressCountry": "India",
"addressLocality": "Mumbai",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Camus"
],
"familyNames": [
"Nimmo"
],
"type": "Person",
"affiliations": [
{
"name": "University College London",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "London",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Nana"
],
"familyNames": [
"Okozi"
],
"type": "Person",
"affiliations": [
{
"name": "National Institute for Communicable Diseases",
"address": {
"addressCountry": "South Africa",
"addressLocality": "Johannesburg",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Rosangela",
"Siqueira"
],
"familyNames": [
"Oliveira"
],
"type": "Person",
"affiliations": [
{
"name": "Institute Adolfo Lutz",
"address": {
"addressCountry": "Brazil",
"addressLocality": "Sao Paulo",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Shaheed",
"Vally"
],
"familyNames": [
"Omar"
],
"type": "Person",
"affiliations": [
{
"name": "National Institute for Communicable Diseases",
"address": {
"addressCountry": "South Africa",
"addressLocality": "Johannesburg",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Nicholas"
],
"familyNames": [
"Paton"
],
"type": "Person",
"affiliations": [
{
"name": "Singapore",
"address": {
"addressCountry": "Singapore",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Timothy",
"EA"
],
"familyNames": [
"Peto"
],
"type": "Person",
"affiliations": [
{
"name": "University of Oxford",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Oxford",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Juliana",
"Maira"
],
"familyNames": [
"Watanabe",
"Pinhata"
],
"type": "Person",
"affiliations": [
{
"name": "Institute Adolfo Lutz",
"address": {
"addressCountry": "Brazil",
"addressLocality": "Sao Paulo",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Sara"
],
"familyNames": [
"Plesnik"
],
"type": "Person",
"affiliations": [
{
"name": "Institute of Microbiology and Laboratory Medicine,WHO-SRL Gauting",
"address": {
"addressCountry": "Germany",
"addressLocality": "IMLred",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Zully",
"M"
],
"familyNames": [
"Puyen"
],
"type": "Person",
"affiliations": [
{
"name": "Instituto Nacional de Salud",
"address": {
"addressCountry": "Peru",
"addressLocality": "Lima",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Marie",
"Sylvianne"
],
"familyNames": [
"Rabodoarivelo"
],
"type": "Person",
"affiliations": [
{
"name": "Institut Pasteur de Madagascar",
"address": {
"addressCountry": "Madagascar",
"addressLocality": "Antananarivo",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Niaina"
],
"familyNames": [
"Rakotosamimanana"
],
"type": "Person",
"affiliations": [
{
"name": "Institut Pasteur de Madagascar",
"address": {
"addressCountry": "Madagascar",
"addressLocality": "Antananarivo",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Paola",
"MV"
],
"familyNames": [
"Rancoita"
],
"type": "Person",
"affiliations": [
{
"name": "Vita-Salute San Raffaele University",
"address": {
"addressCountry": "Italy",
"addressLocality": "Milan",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Priti"
],
"familyNames": [
"Rathod"
],
"type": "Person",
"affiliations": [
{
"name": "UK Health Security Agency",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Birmingham",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Esther"
],
"familyNames": [
"Robinson"
],
"type": "Person",
"affiliations": [
{
"name": "UK Health Security Agency",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Birmingham",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Gillian"
],
"familyNames": [
"Rodger"
],
"type": "Person",
"affiliations": [
{
"name": "University of Oxford",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Oxford",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Camilla"
],
"familyNames": [
"Rodrigues"
],
"type": "Person",
"affiliations": [
{
"name": "Hinduja Hospital",
"address": {
"addressCountry": "India",
"addressLocality": "Mumbai",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Timothy",
"C"
],
"familyNames": [
"Rodwell"
],
"type": "Person",
"affiliations": [
{
"name": "FIND",
"address": {
"addressCountry": "Switzerland",
"addressLocality": "Geneva",
"type": "PostalAddress"
},
"type": "Organization"
},
{
"name": "University of California, San Diego",
"address": {
"addressCountry": "United States",
"addressLocality": "San Diego",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Aysha"
],
"familyNames": [
"Roohi"
],
"type": "Person",
"affiliations": [
{
"name": "University of Oxford",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Oxford",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"David"
],
"familyNames": [
"Santos-Lazaro"
],
"type": "Person",
"affiliations": [
{
"name": "Instituto Nacional de Salud",
"address": {
"addressCountry": "Peru",
"addressLocality": "Lima",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Sanchi"
],
"familyNames": [
"Shah"
],
"type": "Person",
"affiliations": [
{
"name": "The Foundation for Medical Research",
"address": {
"addressCountry": "India",
"addressLocality": "Mumbai",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Thomas",
"Andreas"
],
"familyNames": [
"Kohl"
],
"type": "Person",
"affiliations": [
{
"name": "Research Center Borstel",
"address": {
"addressCountry": "Germany",
"addressLocality": "Borstel",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Grace"
],
"familyNames": [
"Smith"
],
"type": "Person",
"affiliations": [
{
"name": "UK Health Security Agency",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "London",
"type": "PostalAddress"
},
"type": "Organization"
},
{
"name": "UK Health Security Agency",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Birmingham",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Walter"
],
"familyNames": [
"Solano"
],
"type": "Person",
"affiliations": [
{
"name": "Universidad Peruana Cayetano Heredia",
"address": {
"addressCountry": "Peru",
"addressLocality": "Lima",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Andrea"
],
"familyNames": [
"Spitaleri"
],
"type": "Person",
"affiliations": [
{
"name": "IRCCS San Raffaele Scientific Institute",
"address": {
"addressCountry": "Italy",
"addressLocality": "Milan",
"type": "PostalAddress"
},
"type": "Organization"
},
{
"name": "Vita-Salute San Raffaele University",
"address": {
"addressCountry": "Italy",
"addressLocality": "Milan",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Philip"
],
"familyNames": [
"Supply"
],
"type": "Person",
"affiliations": [
{
"name": "Institut Pasteur de Lille",
"address": {
"addressCountry": "France",
"addressLocality": "Lille",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Adrie",
"JC"
],
"familyNames": [
"Steyn"
],
"type": "Person",
"affiliations": [
{
"name": "Africa Health Research Institute",
"address": {
"addressCountry": "South Africa",
"addressLocality": "Durban",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Utkarsha"
],
"familyNames": [
"Surve"
],
"type": "Person",
"affiliations": [
{
"name": "Hinduja Hospital",
"address": {
"addressCountry": "India",
"addressLocality": "Mumbai",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Sabira"
],
"familyNames": [
"Tahseen"
],
"type": "Person",
"affiliations": [
{
"name": "National TB Reference Laboratory, National TB Control Program",
"address": {
"addressCountry": "Pakistan",
"addressLocality": "Islamabad",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Nguyen",
"Thuy"
],
"familyNames": [
"Thuong"
],
"type": "Person",
"affiliations": [
{
"name": "Oxford University Clinical Research Unit",
"address": {
"addressCountry": "Viet Nam",
"addressLocality": "Ho Chi Minh City",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Guy"
],
"familyNames": [
"Thwaites"
],
"type": "Person",
"affiliations": [
{
"name": "University of Oxford",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Oxford",
"type": "PostalAddress"
},
"type": "Organization"
},
{
"name": "Oxford University Clinical Research Unit",
"address": {
"addressCountry": "Viet Nam",
"addressLocality": "Ho Chi Minh City",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Katharina"
],
"familyNames": [
"Todt"
],
"type": "Person",
"affiliations": [
{
"name": "Institute of Microbiology and Laboratory Medicine,WHO-SRL Gauting",
"address": {
"addressCountry": "Germany",
"addressLocality": "IMLred",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Alberto"
],
"familyNames": [
"Trovato"
],
"type": "Person",
"affiliations": [
{
"name": "IRCCS San Raffaele Scientific Institute",
"address": {
"addressCountry": "Italy",
"addressLocality": "Milan",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Christian"
],
"familyNames": [
"Utpatel"
],
"type": "Person",
"affiliations": [
{
"name": "Research Center Borstel",
"address": {
"addressCountry": "Germany",
"addressLocality": "Borstel",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Annelies"
],
"familyNames": [
"Van",
"Rie"
],
"type": "Person",
"affiliations": [
{
"name": "Antwerp",
"address": {
"addressCountry": "Belgium",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Srinivasan"
],
"familyNames": [
"Vijay"
],
"type": "Person",
"affiliations": [
{
"name": "Edinburgh",
"address": {
"addressCountry": "United Kingdom",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Timothy",
"M"
],
"familyNames": [
"Walker"
],
"type": "Person",
"affiliations": [
{
"name": "University of Oxford",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Oxford",
"type": "PostalAddress"
},
"type": "Organization"
},
{
"name": "University of Oxford",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Oxford",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"A",
"Sarah"
],
"familyNames": [
"Walker"
],
"type": "Person",
"affiliations": [
{
"name": "University of Oxford",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Oxford",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Robin"
],
"familyNames": [
"Warren"
],
"type": "Person",
"affiliations": [
{
"name": "Stellenbosch University",
"address": {
"addressCountry": "South Africa",
"addressLocality": "Cape Town",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Jim"
],
"familyNames": [
"Werngren"
],
"type": "Person",
"affiliations": [
{
"name": "Solna",
"address": {
"addressCountry": "Sweden",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Ramona"
],
"familyNames": [
"Groenheit"
],
"type": "Person",
"affiliations": [
{
"name": "Solna",
"address": {
"addressCountry": "Sweden",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Maria"
],
"familyNames": [
"Wijkander"
],
"type": "Person",
"affiliations": [
{
"name": "Solna",
"address": {
"addressCountry": "Sweden",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Robert",
"J"
],
"familyNames": [
"Wilkinson"
],
"type": "Person",
"affiliations": [
{
"name": "Imperial College",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "London",
"type": "PostalAddress"
},
"type": "Organization"
},
{
"name": "Wellcome Centre for Infectious Diseases Research in Africa",
"address": {
"addressCountry": "South Africa",
"addressLocality": "Cape Town",
"type": "PostalAddress"
},
"type": "Organization"
},
{
"name": "Francis Crick Institute",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "London",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Daniel",
"J"
],
"familyNames": [
"Wilson"
],
"type": "Person",
"affiliations": [
{
"name": "University of Oxford",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Oxford",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Penelope"
],
"familyNames": [
"Wintringer"
],
"type": "Person",
"affiliations": [
{
"name": "EMBL-EBI",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Hinxton",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Yu-Xin"
],
"familyNames": [
"Xiao"
],
"type": "Person",
"affiliations": [
{
"name": "Taiwan Centers for Disease Control",
"address": {
"addressCountry": "Taiwan",
"addressLocality": "Taipei",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Yang"
],
"familyNames": [
"Yang"
],
"type": "Person",
"affiliations": [
{
"name": "University of Oxford",
"address": {
"addressCountry": "United Kingdom",
"addressLocality": "Oxford",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Zhao"
],
"familyNames": [
"Yanlin"
],
"type": "Person",
"affiliations": [
{
"name": "Chinese Center for Disease Control and Prevention",
"address": {
"addressCountry": "China",
"addressLocality": "Beijing",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Shen-Yuan"
],
"familyNames": [
"Yao"
],
"type": "Person",
"affiliations": [
{
"name": "National Institute for Communicable Diseases",
"address": {
"addressCountry": "South Africa",
"addressLocality": "Johannesburg",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Baoli"
],
"familyNames": [
"Zhu"
],
"type": "Person",
"affiliations": [
{
"name": "Beijing",
"address": {
"addressCountry": "China",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Stefan"
],
"familyNames": [
"Niemann"
],
"type": "Person",
"affiliations": [
{
"name": "Research Center Borstel",
"address": {
"addressCountry": "Germany",
"addressLocality": "Borstel",
"type": "PostalAddress"
},
"type": "Organization"
},
{
"name": "German Center for Infection Research (DZIF)",
"address": {
"addressCountry": "Germany",
"addressLocality": "Hamburg-Lubeck-Borstel-Riems",
"type": "PostalAddress"
},
"type": "Organization"
}
]
},
{
"givenNames": [
"Max"
],
"familyNames": [
"O'Donnell"
],
"type": "Person",
"affiliations": [
{
"name": "New York",
"address": {
"addressCountry": "United States",
"type": "PostalAddress"
},
"type": "Organization"
}
]
}
],
"type": "Organization"
}
],
"editors": [
{
"givenNames": [
"Bavesh",
"D"
],
"familyNames": [
"Kana"
],
"type": "Person",
"affiliations": [
{
"name": "South Africa",
"type": "Organization"
}
]
}
],
"datePublished": {
"value": "2022-05-19",
"type": "Date"
},
"dateReceived": {
"value": "2021-10-28",
"type": "Date"
},
"dateAccepted": {
"value": "2022-05-15",
"type": "Date"
},
"description": "Tuberculosis is a respiratory disease that is treatable with antibiotics. An increasing prevalence of resistance means that to ensure a good treatment outcome it is desirable to test the susceptibility of each infection to different antibiotics. Conventionally, this is done by culturing a clinical sample and then exposing aliquots to a panel of antibiotics, each being present at a pre-determined concentration, thereby determining if the sample isresistant or susceptible to each sample. The minimum inhibitory concentration (MIC) of a drug is the lowestconcentration that inhibits growth and is a more useful quantity but requires each sample to be tested at a range ofconcentrations for each drug. Using 96-well broth micro dilution plates with each well containing a lyophilised pre-determined amount of an antibiotic is a convenient and cost-effective way to measure the MICs of several drugs at once for a clinical sample. Although accurate, this is still an expensive and slow process that requires highly-skilled and experienced laboratory scientists. Here we show that, through the BashTheBug project hosted on the Zooniverse citizen science platform, a crowd of volunteers can reproducibly and accurately determine the MICs for 13 drugs and that simply taking the median or mode of 11–17 independent classifications is sufficient. There is therefore a potential role for crowds to support (but not supplant) the role of experts in antibiotic susceptibility testing.",
"isPartOf": {
"volumeNumber": 11,
"isPartOf": {
"title": "eLife",
"issns": [
"2050-084X"
],
"identifiers": [
{
"name": "nlm-ta",
"propertyID": "https://registry.identifiers.org/registry/nlm-ta",
"value": "elife",
"type": "PropertyValue"
},
{
"name": "publisher-id",
"propertyID": "https://registry.identifiers.org/registry/publisher-id",
"value": "eLife",
"type": "PropertyValue"
}
],
"publisher": {
"name": "eLife Sciences Publications, Ltd",
"type": "Organization"
},
"type": "Periodical"
},
"type": "PublicationVolume"
},
"licenses": [
{
"url": "http://creativecommons.org/licenses/by/4.0/",
"content": [
{
"content": [
"This article is distributed under the terms of the ",
{
"content": [
"Creative Commons Attribution License"
],
"target": "http://creativecommons.org/licenses/by/4.0/",
"type": "Link"
},
", which permits unrestricted use and redistribution provided that the original author and source are credited."
],
"type": "Paragraph"
}
],
"type": "CreativeWork"
}
],
"keywords": [
"M. tuberculosis",
"citizen science",
"clinical microbiology",
"tuberculosis",
"antibiotics",
"Other"
],
"identifiers": [
{
"name": "publisher-id",
"propertyID": "https://registry.identifiers.org/registry/publisher-id",
"value": 75046,
"type": "PropertyValue"
},
{
"name": "doi",
"propertyID": "https://registry.identifiers.org/registry/doi",
"value": "10.7554/eLife.75046",
"type": "PropertyValue"
},
{
"name": "elocation-id",
"propertyID": "https://registry.identifiers.org/registry/elocation-id",
"value": "e75046",
"type": "PropertyValue"
}
],
"fundedBy": [
{
"identifiers": [
{
"value": "200205/Z/15/Z",
"type": "PropertyValue"
}
],
"funders": [
{
"name": "Wellcome Trust",
"type": "Organization"
}
],
"type": "MonetaryGrant"
},
{
"identifiers": [
{
"value": "OPP1133541",
"type": "PropertyValue"
}
],
"funders": [
{
"name": "Bill and Melinda Gates Foundation",
"type": "Organization"
}
],
"type": "MonetaryGrant"
},
{
"identifiers": [
{
"value": "203141/Z/16/Z",
"type": "PropertyValue"
}
],
"funders": [
{
"name": "Wellcome Trust",
"type": "Organization"
}
],
"type": "MonetaryGrant"
}
],
"about": [
{
"name": "Microbiology and Infectious Disease",
"type": "DefinedTerm"
}
],
"genre": [
"Research Article"
],
"references": [
{
"id": "bib1",
"authors": [
{
"givenNames": [
"J"
],
"familyNames": [
"Cox"
],
"type": "Person"
},
{
"givenNames": [
"EY"
],
"familyNames": [
"Oh"
],
"type": "Person"
},
{
"givenNames": [
"B"
],
"familyNames": [
"Simmons"
],
"type": "Person"
},
{
"givenNames": [
"C"
],
"familyNames": [
"Lintott"
],
"type": "Person"
},
{
"givenNames": [
"K"
],
"familyNames": [
"Masters"
],
"type": "Person"
},
{
"givenNames": [
"A"
],
"familyNames": [
"Greenhill"
],
"type": "Person"
},
{
"givenNames": [
"G"
],
"familyNames": [
"Graham"
],
"type": "Person"
},
{
"givenNames": [
"K"
],
"familyNames": [
"Holmes"
],
"type": "Person"
}
],
"title": "Defining and Measuring Success in Online Citizen Science: A Case Study of Zooniverse Projects",
"datePublished": {
"value": "2015",
"type": "Date"
},
"pageStart": 28,
"pageEnd": 41,
"isPartOf": {
"volumeNumber": 17,
"isPartOf": {
"name": "Computing in Science & Engineering",
"type": "Periodical"
},
"type": "PublicationVolume"
},
"identifiers": [
{
"name": "doi",
"propertyID": "https://registry.identifiers.org/registry/doi",
"value": "10.1109/MCSE.2015.65",
"type": "PropertyValue"
}
],
"type": "Article"
},
{
"id": "bib2",
"authors": [
{
"name": "CRyPTIC Consortium",
"type": "Organization"
}
],
"title": "Epidemiological cutoff values for a 96-well broth microdilution plate for high-throughput research antibiotic susceptibility testing of M. tuberculosis",
"datePublished": {
"value": "2022",
"type": "Date"
},
"isPartOf": {
"volumeNumber": 1949,
"isPartOf": {
"name": "The European Respiratory Journal",
"type": "Periodical"
},
"type": "PublicationVolume"
},
"identifiers": [
{
"name": "doi",
"propertyID": "https://registry.identifiers.org/registry/doi",
"value": "10.1183/13993003.00239-2022",
"type": "PropertyValue"
},
{
"name": "pmid",
"propertyID": "https://registry.identifiers.org/registry/pmid",
"value": 35301246,
"type": "PropertyValue"
}
],
"type": "Article"
},
{
"id": "bib3",
"authors": [
{
"givenNames": [
"PW"
],
"familyNames": [
"Fowler"
],
"type": "Person"
}
],
"title": "Help us fight antibiotic resistance",
"datePublished": {
"value": "2017",
"type": "Date"
},
"url": "https://bashthebug.net/bashthebug-on-the-zooniverse/",
"type": "Article"
},
{
"id": "bib4",
"authors": [
{
"givenNames": [
"PW"
],
"familyNames": [
"Fowler"
],
"type": "Person"
}
],
"title": "pyniverse: a Python package to analyse classifications made by volunteers in a generic Zooniverse citizen science project",
"datePublished": {
"value": "2018",
"type": "Date"
},
"url": "https://github.com/fowler-lab/pyniverse",
"type": "Article"
},
{
"id": "bib5",
"authors": [
{
"givenNames": [
"PW"
],
"familyNames": [
"Fowler"
],
"type": "Person"
}
],
"title": "bashthebug: a Python package to analyse the results of the Zooniverse volunteers for the BashTheBug citizen science project",
"datePublished": {
"value": "2018",
"type": "Date"
},
"url": "https://github.com/fowler-lab/bashthebug",
"type": "Article"
},
{
"id": "bib6",
"authors": [
{
"givenNames": [
"PW"
],
"familyNames": [
"Fowler"
],
"type": "Person"
},
{
"givenNames": [
"AL"
],
"familyNames": [
"Gibertoni",
"Cruz"
],
"type": "Person"
},
{
"givenNames": [
"SJ"
],
"familyNames": [
"Hoosdally"
],
"type": "Person"
},
{
"givenNames": [
"L"
],
"familyNames": [
"Jarrett"
],
"type": "Person"
},
{
"givenNames": [
"E"
],
"familyNames": [
"Borroni"
],
"type": "Person"
},
{
"givenNames": [
"M"
],
"familyNames": [
"Chiacchiaretta"
],
"type": "Person"
},
{
"givenNames": [
"P"
],
"familyNames": [
"Rathod"
],
"type": "Person"
},
{
"givenNames": [
"S"
],
"familyNames": [
"Lehmann"
],
"type": "Person"
},
{
"givenNames": [
"N"
],
"familyNames": [
"Molodtsov"
],
"type": "Person"
},
{
"givenNames": [
"TM"
],
"familyNames": [
"Walker"
],
"type": "Person"
},
{
"givenNames": [
"E"
],
"familyNames": [
"Robinson"
],
"type": "Person"
},
{
"givenNames": [
"H"
],
"familyNames": [
"Hoffmann"
],
"type": "Person"
},
{
"givenNames": [
"TEA"
],
"familyNames": [
"Peto"
],
"type": "Person"
},
{
"givenNames": [
"DM"
],
"familyNames": [
"Cirillo"
],
"type": "Person"
},
{
"givenNames": [
"GE"
],
"familyNames": [
"Smith"
],
"type": "Person"
},
{
"givenNames": [
"DW"
],
"familyNames": [
"Crook"
],
"type": "Person"
}
],
"title": "Automated detection of bacterial growth on 96-well plates for high-throughput drug susceptibility testing of Mycobacterium tuberculosis",
"datePublished": {
"value": "2018",
"type": "Date"
},
"pageStart": 1522,
"pageEnd": 1530,
"isPartOf": {
"volumeNumber": 164,
"isPartOf": {
"name": "Microbiology (Reading, England)",
"type": "Periodical"
},
"type": "PublicationVolume"
},
"identifiers": [
{
"name": "doi",
"propertyID": "https://registry.identifiers.org/registry/doi",
"value": "10.1099/mic.0.000733",
"type": "PropertyValue"
},
{
"name": "pmid",
"propertyID": "https://registry.identifiers.org/registry/pmid",
"value": 30351270,
"type": "PropertyValue"
}
],
"type": "Article"
},
{
"id": "bib7",
"authors": [
{
"name": "Fowler Lab",
"type": "Organization"
}
],
"title": "bashthebug-consensus-dataset",
"datePublished": {
"value": "2022",
"type": "Date"
},
"url": "https://archive.softwareheritage.org/swh:1:dir:c34c01c11f511481b3505e897bad80a091e9cf7a;origin=https://github.com/fowler-lab/bashthebug-consensus-dataset;visit=swh:1:snp:473a0168a5c1542beaf77d3a22fa8bdb8c841a41;anchor=swh:1:rev:61ba253936f7ec33a785158ee48c84e63256cd8b",
"type": "Article"
},
{
"id": "bib8",
"authors": [
{
"givenNames": [
"F"
],
"familyNames": [
"Galton"
],
"type": "Person"
}
],
"title": "Vox populi",
"datePublished": {
"value": "1907",
"type": "Date"
},
"pageStart": 450,
"pageEnd": 451,
"isPartOf": {
"volumeNumber": 1949,
"isPartOf": {
"name": "Nature",
"type": "Periodical"
},
"type": "PublicationVolume"
},
"type": "Article"
},
{
"id": "bib9",
"authors": [
{
"name": "International Standards Organization",
"type": "Organization"
}
],
"title": "ISO 20776-2: Clinical laboratory testing and in vitro diagnostic test systems Technical report",
"datePublished": {
"value": "2007",
"type": "Date"
},
"publisher": {
"name": "International Standards Organization",
"type": "Organization"
},
"type": "Article"
},
{
"id": "bib10",
"authors": [
{
"givenNames": [
"J"
],
"familyNames": [
"Koebel"
],
"type": "Person"
}
],
"title": "Geometrei von K¨unstlichem Feldmessen",
"datePublished": {
"value": "1563",
"type": "Date"
},
"publisher": {
"name": "Egenolff, Franckfurt am Meyn",
"type": "Organization"
},
"type": "Article"
},
{
"id": "bib11",
"authors": [
{
"givenNames": [
"GP"
],
"familyNames": [
"Kubica"
],
"type": "Person"
},
{
"givenNames": [
"TH"
],
"familyNames": [
"Kim"
],
"type": "Person"
},
{
"givenNames": [
"FP"
],
"familyNames": [
"Dunbar"
],
"type": "Person"
}
],
"title": "Designation of Strain H37Rv as the Neotype of Mycobacterium tuberculosis",
"datePublished": {
"value": "1972",
"type": "Date"
},
"pageStart": 99,
"pageEnd": 106,
"isPartOf": {
"volumeNumber": 22,
"isPartOf": {
"name": "International Journal of Systematic Bacteriology",
"type": "Periodical"
},
"type": "PublicationVolume"
},
"identifiers": [
{
"name": "doi",
"propertyID": "https://registry.identifiers.org/registry/doi",
"value": "10.1099/00207713-22-2-99",
"type": "PropertyValue"
}
],
"type": "Article"
},
{
"id": "bib12",
"authors": [
{
"givenNames": [
"N"
],
"familyNames": [
"Littlestone"
],
"type": "Person"
},
{
"givenNames": [
"M"
],
"familyNames": [
"Warmuth"
],
"type": "Person"
}
],
"title": "In 30th Annual Symposium on Foundations of Computer Science",
"datePublished": {
"value": "1989",
"type": "Date"
},
"pageStart": 256,
"pageEnd": 261,
"type": "Article"
},
{
"id": "bib13",
"authors": [
{
"givenNames": [
"W"
],
"familyNames": [
"McKinney"
],
"type": "Person"
}
],
"title": "In Proceedings of the 9th Python in Science Conference",
"datePublished": {
"value": "2010",
"type": "Date"
},
"pageStart": 51,
"pageEnd": 56,
"type": "Article"
},
{
"id": "bib14",
"authors": [
{
"givenNames": [
"A"
],
"familyNames": [
"McMaster"
],
"type": "Person"
},
{
"givenNames": [
"R"
],
"familyNames": [
"Hutchings"
],
"type": "Person"
},
{
"givenNames": [
"C"
],
"familyNames": [
"Allen"
],
"type": "Person"
},
{
"givenNames": [
"Z"
],
"familyNames": [
"Wolfenbarger"
],
"type": "Person"
},
{
"givenNames": [
"H"
],
"familyNames": [
"Dickinson"
],
"type": "Person"
},
{
"givenNames": [
"L"
],
"familyNames": [
"Trouille"
],
"type": "Person"
},
{
"givenNames": [
"C"
],
"familyNames": [
"Johnson"
],
"type": "Person"
}
],
"title": "Panoptes CLI. A command-line interface for Panoptes, the API behind the Zooniverse",
"datePublished": {
"value": "2021",
"type": "Date"
},
"url": "https://github.com/zooniverse/panoptes-cli",
"type": "Article"
},
{
"id": "bib15",
"authors": [
{
"givenNames": [
"J"
],
"familyNames": [
"O’Neill"
],
"type": "Person"
}
],
"title": "Tackling Drug-Resistant Infections Globally: Final Report and Recommendations",
"datePublished": {
"value": "2016",
"type": "Date"
},
"publisher": {
"name": "Technical report",
"type": "Organization"
},
"type": "Article"
},
{
"id": "bib16",
"authors": [
{
"givenNames": [
"LJ"
],
"familyNames": [
"Pankhurst"
],
"type": "Person"
},
{
"givenNames": [
"C"
],
"familyNames": [
"Del",
"Ojo",
"Elias"
],
"type": "Person"
},
{
"givenNames": [
"AA"
],
"familyNames": [
"Votintseva"
],
"type": "Person"
},
{
"givenNames": [
"TM"
],
"familyNames": [
"Walker"
],
"type": "Person"
},
{
"givenNames": [
"K"
],
"familyNames": [
"Cole"
],
"type": "Person"
},
{
"givenNames": [
"J"
],
"familyNames": [
"Davies"
],
"type": "Person"
},
{
"givenNames": [
"JM"
],
"familyNames": [
"Fermont"
],
"type": "Person"
},
{
"givenNames": [
"DM"
],
"familyNames": [
"Gascoyne-Binzi"
],
"type": "Person"
},
{
"givenNames": [
"TA"
],
"familyNames": [
"Kohl"
],
"type": "Person"
},
{
"givenNames": [
"C"
],
"familyNames": [
"Kong"
],
"type": "Person"
},
{
"givenNames": [
"N"
],
"familyNames": [
"Lemaitre"
],
"type": "Person"
},
{
"givenNames": [
"S"
],
"familyNames": [
"Niemann"
],
"type": "Person"
},
{
"givenNames": [
"J"
],
"familyNames": [
"Paul"
],
"type": "Person"
},
{
"givenNames": [
"TR"
],
"familyNames": [
"Rogers"
],
"type": "Person"
},
{
"givenNames": [
"E"
],
"familyNames": [
"Roycroft"
],
"type": "Person"
},
{
"givenNames": [
"EG"
],
"familyNames": [
"Smith"
],
"type": "Person"
},
{
"givenNames": [
"P"
],
"familyNames": [
"Supply"
],
"type": "Person"
},
{
"givenNames": [
"P"
],
"familyNames": [
"Tang"
],
"type": "Person"
},
{
"givenNames": [
"MH"
],
"familyNames": [
"Wilcox"
],
"type": "Person"
},
{
"givenNames": [
"S"
],
"familyNames": [
"Wordsworth"
],
"type": "Person"
},
{
"givenNames": [
"D"
],
"familyNames": [
"Wyllie"
],
"type": "Person"
},
{
"givenNames": [
"L"
],
"familyNames": [
"Xu"
],
"type": "Person"
},
{
"givenNames": [
"DW"
],
"familyNames": [
"Crook"
],
"type": "Person"
},
{
"name": "COMPASS-TB Study Group",
"type": "Organization"
}
],
"title": "Rapid, comprehensive, and affordable mycobacterial diagnosis with whole-genome sequencing: a prospective study",
"datePublished": {
"value": "2016",
"type": "Date"
},
"pageStart": 49,
"pageEnd": 58,
"isPartOf": {
"volumeNumber": 4,
"isPartOf": {
"name": "The Lancet. Respiratory Medicine",
"type": "Periodical"
},
"type": "PublicationVolume"
},
"identifiers": [
{
"name": "doi",
"propertyID": "https://registry.identifiers.org/registry/doi",
"value": "10.1016/S2213-2600(15)00466-X",
"type": "PropertyValue"
},
{
"name": "pmid",
"propertyID": "https://registry.identifiers.org/registry/pmid",
"value": 26669893,
"type": "PropertyValue"
}
],
"type": "Article"
},
{
"id": "bib17",
"authors": [
{
"givenNames": [
"PMV"
],
"familyNames": [
"Rancoita"
],
"type": "Person"
},
{
"givenNames": [
"F"
],
"familyNames": [
"Cugnata"
],
"type": "Person"
},
{
"givenNames": [
"AL"
],
"familyNames": [
"Gibertoni",
"Cruz"
],
"type": "Person"
},
{
"givenNames": [
"E"
],
"familyNames": [
"Borroni"
],
"type": "Person"
},
{
"givenNames": [
"SJ"
],
"familyNames": [
"Hoosdally"
],
"type": "Person"
},
{
"givenNames": [
"TM"
],
"familyNames": [
"Walker"
],
"type": "Person"
},
{
"givenNames": [
"C"
],
"familyNames": [
"Grazian"
],
"type": "Person"
},
{
"givenNames": [
"TJ"
],
"familyNames": [
"Davies"
],
"type": "Person"
},
{
"givenNames": [
"TEA"
],
"familyNames": [
"Peto"
],
"type": "Person"
},
{
"givenNames": [
"DW"
],
"familyNames": [
"Crook"
],
"type": "Person"
},
{
"givenNames": [
"PW"
],
"familyNames": [
"Fowler"
],
"type": "Person"
},
{
"givenNames": [
"DM"
],
"familyNames": [
"Cirillo"
],
"type": "Person"
},
{
"givenNames": [
"DW"
],
"familyNames": [
"Crook"
],
"type": "Person"
},
{
"givenNames": [
"TEA"
],
"familyNames": [
"Peto"
],
"type": "Person"
},
{
"givenNames": [
"AS"
],
"familyNames": [
"Walker"
],
"type": "Person"
},
{
"givenNames": [
"SJ"
],
"familyNames": [
"Hoosdally"
],
"type": "Person"
},
{
"givenNames": [
"AL"
],
"familyNames": [
"Gibertoni",
"Cruz"
],
"type": "Person"
},
{
"givenNames": [
"C"
],
"familyNames": [
"Grazian"
],
"type": "Person"
},
{
"givenNames": [
"TM"
],
"familyNames": [
"Walker"
],
"type": "Person"
},
{
"givenNames": [
"PW"
],
"familyNames": [
"Fowler"
],
"type": "Person"
},
{
"givenNames": [
"D"
],
"familyNames": [
"Wilson"
],
"type": "Person"
},
{
"givenNames": [
"D"
],
"familyNames": [
"Clifton"
],
"type": "Person"
},
{
"givenNames": [
"Z"
],
"familyNames": [
"Iqbal"
],
"type": "Person"
},
{
"givenNames": [
"M"
],
"familyNames": [
"Hunt"
],
"type": "Person"
},
{
"givenNames": [
"EG"
],
"familyNames": [
"Smith"
],
"type": "Person"
},
{
"givenNames": [
"P"
],
"familyNames": [
"Rathod"
],
"type": "Person"
},
{
"givenNames": [
"L"
],
"familyNames": [
"Jarrett"
],
"type": "Person"
},
{
"givenNames": [
"D"
],
"familyNames": [
"Matias"
],
"type": "Person"
},
{
"givenNames": [
"DM"
],
"familyNames": [
"Cirillo"
],
"type": "Person"
},
{
"givenNames": [
"E"
],
"familyNames": [
"Borroni"
],
"type": "Person"
},
{
"givenNames": [
"S"
],
"familyNames": [
"Battaglia"
],
"type": "Person"
},
{
"givenNames": [
"M"
],
"familyNames": [
"Chiacchiaretta"
],
"type": "Person"
},
{
"givenNames": [
"M"
],
"familyNames": [
"De",
"Filippo"
],
"type": "Person"
},
{
"givenNames": [
"A"
],
"familyNames": [
"Cabibbe"
],
"type": "Person"
},
{
"givenNames": [
"S"
],
"familyNames": [
"Tahseen"
],
"type": "Person"
},
{
"givenNames": [
"N"
],
"familyNames": [
"Mistry"
],
"type": "Person"
},
{
"givenNames": [
"K"
],
"familyNames": [
"Nilgiriwala"
],
"type": "Person"
},
{
"givenNames": [
"V"
],
"familyNames": [
"Chitalia"
],
"type": "Person"
},
{
"givenNames": [
"N"
],
"familyNames": [
"Ganesan"
],
"type": "Person"
},
{
"givenNames": [
"A"
],
"familyNames": [
"Papewar"
],
"type": "Person"
},
{
"givenNames": [
"C"
],
"familyNames": [
"Rodrigues"
],
"type": "Person"
},
{
"givenNames": [
"P"
],
"familyNames": [
"Kambli"
],
"type": "Person"
},
{
"givenNames": [
"U"
],
"familyNames": [
"Surve"
],
"type": "Person"
},
{
"givenNames": [
"R"
],
"familyNames": [
"Khot"
],
"type": "Person"
},
{
"givenNames": [
"S"
],
"familyNames": [
"Niemann"
],
"type": "Person"
},
{
"givenNames": [
"T"
],
"familyNames": [
"Kohl"
],
"type": "Person"
},
{
"givenNames": [
"M"
],
"familyNames": [
"Merker"
],
"type": "Person"
},
{
"givenNames": [
"H"
],
"familyNames": [
"Hoffmann"
],
"type": "Person"
},
{
"givenNames": [
"S"
],
"familyNames": [
"Lehmann"
],
"type": "Person"
},
{
"givenNames": [
"S"
],
"familyNames": [
"Plesnik"
],
"type": "Person"
},
{
"givenNames": [
"N"
],
"familyNames": [
"Ismail"
],
"type": "Person"
},
{
"givenNames": [
"SV"
],
"familyNames": [
"Omar"
],
"type": "Person"
},
{
"givenNames": [
"L"
],
"familyNames": [
"Joseph"
],
"type": "Person"
},
{
"givenNames": [
"E"
],
"familyNames": [
"Marubini"
],
"type": "Person"
},
{
"givenNames": [
"G"
],
"familyNames": [
"Thwaites"
],
"type": "Person"
},
{
"givenNames": [
"TN"
],
"familyNames": [
"Thuy",
"Thuong"
],
"type": "Person"
},
{
"givenNames": [
"NH"
],
"familyNames": [
"Ngoc"
],
"type": "Person"
},
{
"givenNames": [
"V"
],
"familyNames": [
"Srinivasan"
],
"type": "Person"
},
{
"givenNames": [
"D"
],
"familyNames": [
"Moore"
],
"type": "Person"
},
{
"givenNames": [
"J"
],
"familyNames": [
"Coronel"
],
"type": "Person"
},
{
"givenNames": [
"W"
],
"familyNames": [
"Solano"
],
"type": "Person"
},
{
"givenNames": [
"G"
],
"familyNames": [
"He"
],
"type": "Person"
},
{
"givenNames": [
"B"
],
"familyNames": [
"Zhu"
],
"type": "Person"
},
{
"givenNames": [
"Y"
],
"familyNames": [
"Zhou"
],
"type": "Person"
},
{
"givenNames": [
"A"
],
"familyNames": [
"Ma"
],
"type": "Person"
},
{
"givenNames": [
"P"
],
"familyNames": [
"Yu"
],
"type": "Person"
},
{
"givenNames": [
"M"
],
"familyNames": [
"Schito"
],
"type": "Person"
},
{
"givenNames": [
"P"
],
"familyNames": [
"Claxton"
],
"type": "Person"
},
{
"givenNames": [
"I"
],
"familyNames": [
"Laurenson"
],
"type": "Person"
}
],
"title": "Antimicrobial Agents and Chemotherapy",
"datePublished": {
"value": "2018",
"type": "Date"
},
"isPartOf": {
"volumeNumber": 62,
"isPartOf": {
"name": "ASM Journals Logo",
"type": "Periodical"
},
"type": "PublicationVolume"
},
"type": "Article"
},
{
"id": "bib18",
"authors": [
{
"givenNames": [
"T"
],
"familyNames": [
"Schön"
],
"type": "Person"
},
{
"givenNames": [
"E"
],
"familyNames": [
"Matuschek"
],
"type": "Person"
},
{
"givenNames": [
"S"
],
"familyNames": [
"Mohamed"
],
"type": "Person"
},
{
"givenNames": [
"M"
],
"familyNames": [
"Utukuri"
],
"type": "Person"
},
{
"givenNames": [
"S"
],
"familyNames": [
"Heysell"
],
"type": "Person"
},
{
"givenNames": [
"J-W"
],
"familyNames": [
"Alffenaar"
],
"type": "Person"
},
{
"givenNames": [
"S"
],
"familyNames": [
"Shin"
],
"type": "Person"
},
{
"givenNames": [
"E"
],
"familyNames": [
"Martinez"
],
"type": "Person"
},
{
"givenNames": [
"V"
],
"familyNames": [
"Sintchenko"
],
"type": "Person"
},
{
"givenNames": [
"FP"
],
"familyNames": [
"Maurer"
],
"type": "Person"
},
{
"givenNames": [
"PM"
],
"familyNames": [
"Keller"
],
"type": "Person"
},
{
"givenNames": [
"G"
],
"familyNames": [
"Kahlmeter"
],
"type": "Person"
},
{
"givenNames": [
"CU"
],
"familyNames": [
"Köser"
],
"type": "Person"
}
],
"title": "Standards for MIC testing that apply to the majority of bacterial pathogens should also be enforced for Mycobacterium tuberculosis complex",
"datePublished": {
"value": "2019",
"type": "Date"
},
"pageStart": 403,
"pageEnd": 405,
"isPartOf": {
"volumeNumber": 25,
"isPartOf": {
"name": "Clinical Microbiology and Infection",
"type": "Periodical"
},
"type": "PublicationVolume"
},
"identifiers": [
{
"name": "doi",
"propertyID": "https://registry.identifiers.org/registry/doi",
"value": "10.1016/j.cmi.2019.01.019",
"type": "PropertyValue"
},
{
"name": "pmid",
"propertyID": "https://registry.identifiers.org/registry/pmid",
"value": 30771527,
"type": "PropertyValue"
}
],
"type": "Article"
},
{
"id": "bib19",
"authors": [
{
"givenNames": [
"T"
],
"familyNames": [
"Schön"
],
"type": "Person"
},
{
"givenNames": [
"J"
],
"familyNames": [
"Werngren"
],
"type": "Person"
},
{
"givenNames": [
"D"
],
"familyNames": [
"Machado"
],
"type": "Person"
},
{
"givenNames": [
"E"
],
"familyNames": [
"Borroni"
],
"type": "Person"
},
{
"givenNames": [
"M"
],
"familyNames": [
"Wijkander"
],
"type": "Person"
},
{
"givenNames": [
"G"
],
"familyNames": [
"Lina"
],
"type": "Person"
},
{
"givenNames": [
"J"
],
"familyNames": [
"Mouton"
],
"type": "Person"
},
{
"givenNames": [
"E"
],
"familyNames": [
"Matuschek"
],
"type": "Person"
},
{
"givenNames": [
"G"
],
"familyNames": [
"Kahlmeter"
],
"type": "Person"
},
{
"givenNames": [
"C"
],
"familyNames": [
"Giske"
],
"type": "Person"
},
{
"givenNames": [
"M"
],
"familyNames": [
"Santin"
],
"type": "Person"
},
{
"givenNames": [
"DM"
],
"familyNames": [
"Cirillo"
],
"type": "Person"
},
{
"givenNames": [
"M"
],
"familyNames": [
"Viveiros"
],
"type": "Person"
},
{
"givenNames": [
"E"
],
"familyNames": [
"Cambau"
],
"type": "Person"
}
],
"title": "Antimicrobial susceptibility testing of Mycobacterium tuberculosis complex isolates – the EUCAST broth microdilution reference method for MIC determination",
"datePublished": {
"value": "2020",
"type": "Date"
},
"pageStart": 10,
"pageEnd": 13,
"isPartOf": {
"volumeNumber": 27,
"isPartOf": {
"name": "Clinical Microbiology and Infection",
"type": "Periodical"
},
"type": "PublicationVolume"
},
"identifiers": [
{
"name": "doi",
"propertyID": "https://registry.identifiers.org/registry/doi",
"value": "10.1016/j.cmi.2020.10.019",
"type": "PropertyValue"
},
{
"name": "pmid",
"propertyID": "https://registry.identifiers.org/registry/pmid",
"value": 33198949,
"type": "PropertyValue"
}
],
"type": "Article"
},
{
"id": "bib20",
"authors": [
{
"givenNames": [
"H"
],
"familyNames": [
"Spiers"
],
"type": "Person"
},
{
"givenNames": [
"A"
],
"familyNames": [
"Swanson"
],
"type": "Person"
},
{
"givenNames": [
"L"
],
"familyNames": [
"Fortson"
],
"type": "Person"
},
{
"givenNames": [
"BD"
],
"familyNames": [
"Simmons"
],
"type": "Person"
},
{
"givenNames": [
"L"
],
"familyNames": [
"Trouille"
],
"type": "Person"
},
{
"givenNames": [
"S"
],
"familyNames": [
"Blickhan"
],
"type": "Person"
},
{
"givenNames": [
"C"
],
"familyNames": [
"Lintott"
],
"type": "Person"
}
],
"title": "Everyone counts? Design considerations in online citizen science",
"datePublished": {
"value": "2019",
"type": "Date"
},
"isPartOf": {
"volumeNumber": 18,
"isPartOf": {
"name": "Journal of Science Communication",
"type": "Periodical"
},
"type": "PublicationVolume"
},
"identifiers": [
{
"name": "doi",
"propertyID": "https://registry.identifiers.org/registry/doi",
"value": "10.22323/2.18010204",
"type": "PropertyValue"
}
],
"type": "Article"
},
{
"id": "bib21",
"authors": [
{
"givenNames": [
"H"
],
"familyNames": [
"Spiers"
],
"type": "Person"
},
{
"givenNames": [
"H"
],
"familyNames": [
"Songhurst"
],
"type": "Person"
},
{
"givenNames": [
"L"
],
"familyNames": [
"Nightingale"
],
"type": "Person"
},
{
"givenNames": [
"J"
],
"familyNames": [
"de",
"Folter"
],
"type": "Person"
},
{
"givenNames": [
"R"
],
"familyNames": [
"Hutchings"
],
"type": "Person"
},
{
"givenNames": [
"CJ"
],
"familyNames": [
"Peddie"
],
"type": "Person"
},
{
"givenNames": [
"A"
],
"familyNames": [
"Weston"
],
"type": "Person"
},
{
"givenNames": [
"A"
],
"familyNames": [
"Strange"
],
"type": "Person"
},
{
"givenNames": [
"S"
],
"familyNames": [
"Hindmarsh"
],
"type": "Person"
},
{
"givenNames": [
"C"
],
"familyNames": [
"Lintott"
],
"type": "Person"
},
{
"givenNames": [
"LM"
],
"familyNames": [
"Collinson"
],
"type": "Person"
},
{
"givenNames": [
"ML"
],
"familyNames": [
"Jones"
],
"type": "Person"
}
],
"title": "Citizen Science, Cells and CNNs – Deep Learning for Automatic Segmentation of the Nuclear Envelope in Electron Microscopy Data, Trained with Volunteer Segmentations",
"datePublished": {
"value": "2020",
"type": "Date"
},
"isPartOf": {
"name": "bioRxiv",
"type": "Periodical"
},
"identifiers": [
{
"name": "doi",
"propertyID": "https://registry.identifiers.org/registry/doi",
"value": "10.1101/2020.07.28.223024",
"type": "PropertyValue"
}
],
"type": "Article"
},
{
"id": "bib22",
"authors": [
{
"name": "The CRyPTIC Consortium",
"type": "Organization"
}
],
"title": "A Crowd of BashTheBug Volunteers Reproducibly and Accurately Measure the Minimum Inhibitory Concentrations of 13 Antitubercular Drugs from Photographs of 96-Well Broth Microdilution Plates",
"datePublished": {
"value": "2021",
"type": "Date"
},
"isPartOf": {
"name": "bioRxiv",
"type": "Periodical"
},
"identifiers": [
{
"name": "doi",
"propertyID": "https://registry.identifiers.org/registry/doi",
"value": "10.1101/20210914460353",
"type": "PropertyValue"
}
],
"type": "Article"
},
{
"id": "bib23",
"authors": [
{
"name": "The CRyPTIC Consortium",
"type": "Organization"
}
],
"title": "BashTheBug: A Crowd of Volunteers Reproducibly and Accurately Measure the Minimum Inhibitory Concentrations of 13 Antitubercular Drugs from Photographs of 96-Well Broth Microdilution Plates",
"datePublished": {
"value": "2022",
"type": "Date"
},
"isPartOf": {
"name": "bioRxiv",
"type": "Periodical"
},
"identifiers": [
{
"name": "doi",
"propertyID": "https://registry.identifiers.org/registry/doi",
"value": "10.1101/20210914460274",
"type": "PropertyValue"
}
],
"type": "Article"
},
{
"id": "bib24",
"authors": [
{
"givenNames": [
"L"
],
"familyNames": [
"Trouille"
],
"type": "Person"
},
{
"givenNames": [
"CJ"
],
"familyNames": [
"Lintott"
],
"type": "Person"
},
{
"givenNames": [
"LF"
],
"familyNames": [
"Fortson"
],
"type": "Person"
}
],
"title": "Citizen science frontiers: Efficiency, engagement, and serendipitous discovery with human-machine systems",
"datePublished": {
"value": "2019",
"type": "Date"
},
"pageStart": 1902,
"pageEnd": 1909,
"isPartOf": {
"volumeNumber": 116,
"isPartOf": {
"name": "PNAS",
"type": "Periodical"
},
"type": "PublicationVolume"
},
"identifiers": [
{
"name": "doi",
"propertyID": "https://registry.identifiers.org/registry/doi",
"value": "10.1073/pnas.1807190116",
"type": "PropertyValue"
},
{
"name": "pmid",
"propertyID": "https://registry.identifiers.org/registry/pmid",
"value": 30718393,
"type": "PropertyValue"
}
],
"type": "Article"
},
{
"id": "bib25",
"authors": [
{
"givenNames": [
"TM"
],
"familyNames": [
"Walker"
],
"type": "Person"
},
{
"givenNames": [
"ALG"
],
"familyNames": [
"Cruz"
],
"type": "Person"
},
{
"givenNames": [
"TE"
],
"familyNames": [
"Peto"
],
"type": "Person"
},
{
"givenNames": [
"EG"
],
"familyNames": [
"Smith"
],
"type": "Person"
},
{
"givenNames": [
"H"
],
"familyNames": [
"Esmail"
],
"type": "Person"
},
{
"givenNames": [
"DW"
],
"familyNames": [
"Crook"
],
"type": "Person"
}
],
"title": "Tuberculosis is changing",
"datePublished": {
"value": "2017",
"type": "Date"
},
"pageStart": 359,
"pageEnd": 361,
"isPartOf": {
"volumeNumber": 17,
"isPartOf": {
"name": "The Lancet. Infectious Diseases",
"type": "Periodical"
},
"type": "PublicationVolume"
},
"identifiers": [
{
"name": "doi",
"propertyID": "https://registry.identifiers.org/registry/doi",
"value": "10.1016/S1473-3099(17)30123-8",
"type": "PropertyValue"
},
{
"name": "pmid",
"propertyID": "https://registry.identifiers.org/registry/pmid",
"value": 28298254,
"type": "PropertyValue"
}
],
"type": "Article"
},
{
"id": "bib26",
"authors": [
{
"givenNames": [
"TM"
],
"familyNames": [
"Walker"
],
"type": "Person"
},
{
"givenNames": [
"P"
],
"familyNames": [
"Miotto"
],
"type": "Person"
},
{
"givenNames": [
"CU"
],
"familyNames": [
"Köser"
],
"type": "Person"
},
{
"givenNames": [
"PW"
],
"familyNames": [
"Fowler"
],
"type": "Person"
},
{
"givenNames": [
"J"
],
"familyNames": [
"Knaggs"
],
"type": "Person"
},
{
"givenNames": [
"Z"
],
"familyNames": [
"Iqbal"
],
"type": "Person"
},
{
"givenNames": [
"M"
],
"familyNames": [
"Hunt"
],
"type": "Person"
},
{
"givenNames": [
"L"
],
"familyNames": [
"Chindelevitch"
],
"type": "Person"
},
{
"givenNames": [
"M"
],
"familyNames": [
"Farhat"
],
"type": "Person"
},
{
"givenNames": [
"DM"
],
"familyNames": [
"Cirillo"
],
"type": "Person"
},
{
"givenNames": [
"I"
],
"familyNames": [
"Comas"
],
"type": "Person"
},
{
"givenNames": [
"J"
],
"familyNames": [
"Posey"
],
"type": "Person"
},
{
"givenNames": [
"SV"
],
"familyNames": [
"Omar"
],
"type": "Person"
},
{
"givenNames": [
"TE"
],
"familyNames": [
"Peto"
],
"type": "Person"
},
{
"givenNames": [
"A"
],
"familyNames": [
"Suresh"
],
"type": "Person"
},
{
"givenNames": [
"S"
],
"familyNames": [
"Uplekar"
],
"type": "Person"
},
{
"givenNames": [
"S"
],
"familyNames": [
"Laurent"
],
"type": "Person"
},
{
"givenNames": [
"RE"
],
"familyNames": [
"Colman"
],
"type": "Person"
},
{
"givenNames": [
"C-M"
],
"familyNames": [
"Nathanson"
],
"type": "Person"
},
{
"givenNames": [
"M"
],
"familyNames": [
"Zignol"
],
"type": "Person"
},
{
"givenNames": [
"AS"
],
"familyNames": [
"Walker"
],
"type": "Person"
},
{
"givenNames": [
"DW"
],
"familyNames": [
"Crook"
],
"type": "Person"
},
{
"givenNames": [
"N"
],
"familyNames": [
"Ismail"
],
"type": "Person"
},
{
"givenNames": [
"TC"
],
"familyNames": [
"Rodwell"
],
"type": "Person"
},
{
"name": "CRyPTIC Consortium",
"type": "Organization"
},
{
"name": "Seq&Treat Consortium",
"type": "Organization"
}
],
"title": "The 2021 WHO catalogue of Mycobacterium tuberculosis complex mutations associated with drug resistance: A genotypic analysis",
"datePublished": {
"value": "2022",
"type": "Date"
},
"isPartOf": {
"volumeNumber": 3,
"isPartOf": {
"name": "The Lancet. Microbe",
"type": "Periodical"
},
"type": "PublicationVolume"
},
"identifiers": [
{
"name": "doi",
"propertyID": "https://registry.identifiers.org/registry/doi",
"value": "10.1016/S2666-5247(21)00301-3",
"type": "PropertyValue"
},
{
"name": "pmid",
"propertyID": "https://registry.identifiers.org/registry/pmid",
"value": 35373160,
"type": "PropertyValue"
}
],
"type": "Article"
},
{
"id": "bib27",
"authors": [
{
"name": "World Health Organization",
"type": "Organization"
}
],
"title": "Global Tuberculosis Report, Technical report",
"datePublished": {
"value": "2020",
"type": "Date"
},
"publisher": {
"name": "WHO",
"type": "Organization"
},
"type": "Article"
},
{
"id": "bib28",
"authors": [
{
"givenNames": [
"T"
],
"familyNames": [
"Zhu"
],
"type": "Person"
},
{
"givenNames": [
"AEW"
],
"familyNames": [
"Johnson"
],
"type": "Person"
},
{
"givenNames": [
"J"
],
"familyNames": [
"Behar"
],
"type": "Person"
},
{
"givenNames": [
"GD"
],
"familyNames": [
"Clifford"
],
"type": "Person"
}
],
"title": "Crowd-sourced annotation of ecg signals using contextual information",
"datePublished": {
"value": "2014",
"type": "Date"
},
"pageStart": 871,
"pageEnd": 884,
"isPartOf": {
"volumeNumber": 42,
"isPartOf": {
"name": "Annals of Biomedical Engineering",
"type": "Periodical"
},
"type": "PublicationVolume"
},
"identifiers": [
{
"name": "doi",
"propertyID": "https://registry.identifiers.org/registry/doi",
"value": "10.1007/s10439-013-0964-6",
"type": "PropertyValue"
},
{
"name": "pmid",
"propertyID": "https://registry.identifiers.org/registry/pmid",
"value": 24368593,
"type": "PropertyValue"
}
],
"type": "Article"
}
]
},
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Introduction\n",
"\n",
"Tuberculosis (TB) is a treatable (primarily) respiratory disease that caused illness in ten million people in 2019, with 1.4million deaths [@bib27]. Ordinarily this is more than any other single pathogen, although SARS-CoV-2 killed more people than TB in 2020 and is likely to do so again in 2021. Like most bacterial diseases treated with antibiotics, an increasing proportion of TB cases are resistant to one or more drugs.\n",
"\n",
"Tackling this ‘silent pandemic’ will require action on several fronts, including the development of new antibiotics, better stewardship of existing antibiotics and much wider use of antibiotic susceptibility testing (AST) to guide prescribing decisions [@bib15]. The prevailing AST paradigm is _culture-based_: a sample taken from the patient is grown and the pathogen identified. If required, further samples are cultured in the presence of different antibiotics and each test is inspected/measured to see which compounds inhibit the growth of the bacterium. A scientific laboratory, with an enhanced biosafety level and staffed by experienced and highly trained laboratory scientists, is required to carry out such AST. Compared to most pathogenic bacteria, _M. tuberculosis_ is unusually difficult,mainly because its doubling time is so slow resulting in culture times being weeks rather than days. Maintaining such laboratories with a cadre of expert scientists is expensive and hence they tend to be found only at larger hospitals and national public health agencies, even in high-income countries. This model, whilst effective, is practically and economically difficult to scale up which explains in part why most antibiotic prescribing decisions are still done without any AST data.\n",
"\n",
"The _minimum inhibitory concentration_ (MIC) of an antibiotic is the lowest concentration that prevents growth of the pathogen. Mycobacterial samples are usually inoculated onto an appropriate growth medium that contains an antibiotic at a single critical concentration; if the MIC of the same is above the critical concentration it will grow and hence be labelled resistant, otherwise there is no growth and it will be classified as susceptible. Although simple, the accuracy of such an approach decreases when the expected MIC is of the same order as the critical concentration, as is the case for many antituberculars [@bib2]. Historically, it has been assumed that, since accuracy is paramount, only highly-trained and experienced laboratory scientists (experts), or more recently, extensively-validated automatic algorithms part of accredited AST devices, can assess if a pathogen is growing and hence determine whether the sample is resistant or not.\n",
"\n",
"In this paper, we shall show that a crowd of volunteers, who have no microbiological training, can reproducibly and accurately determine the growth of _M. tuberculosis_ (the causative agent of TB) on a 96-well plate and thence the MICs for 13 different antibiotics. This is an application of the “wisdom of crowds” which can be traced back to observations made by [@bib8]. An even earlier use is the Germanic definition of the foot as one sixteenth the combined length of the left feet of the first sixteen men to leave a church on a Sunday [@bib10]. The main goal of the BashTheBug citizen science project, which was launched in April 2017 on the Zooniverse platform, is to reduce MIC measurement error in the large dataset of 20,637 clinical _M. tuberculosis_ isolates collected by the Comprehensive Resistance Prediction for Tuberculosis: an International Consortium (CRyPTIC) project.\n",
"\n",
"# Results\n",
"\n",
"## Project launch and progress\n",
"\n",
"After a successful beta-test on 22 March 2017, BashTheBug was launched as a project on the Zooniverse citizen science platform on 8 April 2017. The launch was publicised on social media, mainly Twitter, and mentioned on several websites and the Zooniverse users were notified via email. By the end of the first week, 2029 people had participated (of which 1,259 had usernames) classifying a total of 74,949 images – this includes the beta-test. The initial set of images were completed on 11 October 2017 and a second set was classified between 8 June 2020 and 15 November 2020 ([Figure 1A](#fig1)) to increase the number of classifications per drug image from 11 to 17.\n",
"\n",
"figure: Figure 1.\n",
":::\n",
"\n",
"\n",
"### This dataset of 778,202 classifications was collected in two batches between April 2017 and Sep 2020 by 9372 volunteers.\n",
"\n",
"(**A**) The classifications were done by the volunteers in two distinct batches; one during 2017 and a later one in 2020. Note that the higher participation during 2020 was due to the national restrictions imposed due to the SARS-Cov-2 pandemic. (**B**) The number of active users per day varied from zero to over 150. (C) The Lorenz curve demonstrates that there is considerable participation inequality in the project resulting in a Gini-coefficient of 0.85. (D) Volunteers spent different lengths of time classifying drug images after 14 days of incubation with a mode duration of 3.5 s.\n",
":::\n",
"{#fig1}\n",
"\n",
"figure: Figure 1—figure supplement 1.\n",
":::\n",
"\n",
"\n",
"### Thank you to all the volunteers who contributed one or more classifications to this manuscript.\n",
"\n",
"There are the 5810 usernames of all the volunteers in this montage – volunteers who did not register or sign in are not included.\n",
":::\n",
"{#fig1s1}\n",
"\n",
"figure: Figure 1—figure supplement 2.\n",
":::\n",
"\n",
"\n",
"### The time spent by volunteers on each classification varied with a mode of 3.5 s.\n",
"\n",
"Since one would expect different amounts of bacterial growth on the microdilution plates after (**A**) 7, (**B**) 10, (**C**) 14 and (**D**) 21 days the distributions of these were examined separately. All were, however, similar indicating that this did not have a significant effect.\n",
":::\n",
"{#fig1s2}\n",
"\n",
"figure: Figure 1—figure supplement 3.\n",
":::\n",
"\n",
"\n",
"### The time spent by volunteers on each classification varied depending on the drug being considered.\n",
"\n",
"The mode of each distribution is labelled. The drug the volunteers spent the longest on (bedaquiline, mode 4.8 s) was also one of those with the largest number (8) of wells. As measured by its mode of 3.2 s, the volunteers spent the least time classifying delamanid.\n",
":::\n",
"{#fig1s3}\n",
"\n",
"figure: Figure 1—figure supplement 4.\n",
":::\n",
"\n",
"\n",
"### Every new user is shown this tutorial when they first join the BashTheBug Zooniverse project.\n",
"\n",
"It uses example images to explain the task and then each of the options that they can choose to classify a drug image.\n",
":::\n",
"{#fig1s4}\n",
"\n",
"## Volunteers\n",
"\n",
"In total, 9372 volunteers participated in classifying this dataset doing a total of 778,202 classifications ([Figure 1—figure supplement 1](#fig1s1)). The number of citizen scientists is an over-estimate since users who did not register with the Zooniverse (and therefore could not be identified through their unique username) but did more than one session will be counted multiple times. On 251 of the 356 days (70 %) when these data were being collected there were ten or more volunteers active on the project ([Figure 1B](#fig1)), and on 31% of the days 50 or more volunteers were active. This illustrates the importance of engaging with the volunteers via multiple channels to ensure an active and sustainable project. Overall, this is a mean of 84.1 classifications per volunteer; however, this hides a large amount of variation in the number of classifications done by individual volunteers. Almost half of the volunteers (4299) did ten or fewer classifications and 1115 classified only a single image whilst the ten volunteers who participated the most did 103,569 classifications between them which is 13.1% of the total. The Gini-coefficient is a useful way to measure these unequal levels of participation, and for this dataset it is 0.85 ([Figure 1C](#fig1)).\n",
"\n",
"## Comparison to other Zooniverse projects\n",
"\n",
"The activity within the first 100 days of launch has been used to benchmark and compare different Zooniverse projects from several academic disciplines [@bib21]. A total of 381,964 classifications were done in the first hundred days after launch by a total of 6237 users of which 3733 were registered and so were unique. Several Zooniverse projects have attracted many more users and classifications; however, these are all ecology or astronomy projects which are the mainstay of the Zooniverse.\n",
"\n",
"Since the number of classifications is heavily influenced by the difficulty of the task, it can be more illuminating to compare the Gini coefficients of different projects. [@bib1] measured a mean Gini coefficient across several Zooniverse projects of 0.81, whilst a later and more comprehensive study [@bib20] demonstrated that Zooniverse projects had Gini coefficients in the range 0.54–0.94 with a mean of 0.80. They also suggested that biomedical projects had lower Gini coefficients, with a mean Gini coefficient of 0.67; however, this was only based on three projects. BashTheBug attracted more users, completed more classifications and had a higher Gini coefficient than any of these three biomedical projects [@bib20]. A more recent biomedical project, Etch-a-cell, that launched at a similar time to BashTheBug had a Gini coefficient of 0.83 [@bib21]. BashTheBug therefore has a higher than average level of participation inequality, having the 17th highest Gini coefficient out of 63 Zooniverse projects surveyed [@bib20].\n",
"\n",
"## Time spent\n",
"\n",
"The time spent by a volunteer classifying a single drug image varied from a few seconds up to hours; the latter are assumed to be data artefacts, for example when a volunteer leaves a browser window open. The distribution of time spent per image shows no appreciable differences when calculated as function of the incubation time with a mode of 3.5 s ([Figure 1D](#fig1), [Figure 1—figure supplement 2](#fig1s2)), which is unexpected given after only 7 days of incubation there is little or no bacterial growth. After 14 days incubation there are, however, observable differences between how long the volunteers spent classifying each drug ([Figure 1—figure supplement 3](#fig1s3)).\n",
"\n",
"## Classification validity\n",
"\n",
"The tutorial on the Zooniverse website ([Figure 1—figure supplement 4](#fig1s4)) encouraged volunteers to check that the control wells both contain bacterial growth – if not then the drug image should be marked as having ‘No Growth in either of the ‘No Antibiotic’ wells’. They were also asked to check if any of the drug wells contain growth very different to all the others (contamination), inconsistent growth (skip wells), or anything else that would prevent a measurement being taken (artefacts). If any of these are true, they were asked to mark the drug image as “Cannot classify”. In the analysis these were aggregated into a single dilution (NR – not read). In all cases, if a simple majority make a classification of NR, then this is always returned as the result. All NR results are excluded from calculations of the exact or essential agreement.\n",
"\n",
"## Expert measurements\n",
"\n",
"Each drug image was also measured by a laboratory scientist using a Vizion instrument as well as programmatically by some software, AMyGDA [@bib3]. Although the AMyGDA software is reproducible, it will often classify artefacts, such as air bubbles, contamination, shadows and sediment, as bacterial growth and is also likely to assess a well as containing no bacterial growth when the level of growth is very low. By contrast, laboratory scientists are less consistent but can recognise and ignore artefacts. Since the sources of error for each these methods are different, we constructed a consensus dataset with an assumed reduced error rate by only including readings where both of these independent methods agree on the value of the MIC. We will refer to this as the ‘Expert +AMyGDA’ dataset and the larger dataset simply consisting of all the readings taken by the laboratory scientist as the ‘Expert’ dataset. We shall further assume the error-rate in the ‘Expert +AMyGDA’ consensus dataset is negligible, allowing us to use it as a reference dataset, which in turn will allow us to infer the performance of the volunteers by comparison.\n",
"\n",
"A total of 12,488 drug images were read after 14 days incubation ([Supplementary file 1](#supp1)); for 6205 (49.7%) of these both the laboratory scientist (Expert) and the software (AMyGDA) returned the same MIC. Since a laboratory scientist would be reasonably expected to make an error ≤ 5% of the time, the majority of the drug images excluded are most likely due to AMyGDA incorrectly reading a drug image with only a minority being genuine errors.\n",
"\n",
"By constructing the consensus Expert+AMyGDA dataset we are likely to have introduced bias by unwittingly selecting for samples which are easier to read. One possibility is that we may have selected samples with higher than average levels of bacterial growth. We can show that this is not the case since not only is the average level of growth in the positive controls (as measured by AMyGDA) for the Expert+AMyGDA dataset (30.4%) similar to that observed (30.6%) for the larger Expert dataset ([Figure 6—figure supplement 3](#fig6s3)), but the distributions themselves are very similar.\n",
"\n",
"A second possibility is that drug images with specific growth configurations (for example either no growth or growth in all the wells) are easier to read than drug images where the growth halts. This would imply that the probability of the Expert+AMyGDA measurements agreeing is a function of the dilution MIC, which indeed is what we find ([Supplementary file 1d](#supp1)). The agreement is highest when there is no growth in any of the drug wells, which makes sense as that is a relatively trivial classification to make. The next highest value is when the dilution is 8, which since 7 of the 14 drugs on the plate have 7 drug wells (Figure 6), corresponds to growth in all 7 drug wells, which is also an easy classification.\n",
"\n",
"The net effect of this is that the Expert+AMyGDA dataset has a different distribution of measured MICs, including a greater proportion of drug images with a low MIC dilution (61.4% after 14 days incubation have a dilution of 1 or 2, [Figure 6—figure supplement 4](#fig6s4)) compared to the parent Expert dataset (45.8%). One should bear in mind this bias when interpreting the results, and to assist we will consider if key results change when we use the Expert-only dataset.\n",
"\n",
"## How to compare?\n",
"\n",
"Ideally one would apply an international standard for antibiotic susceptibility testing (AST) for Mycobacteria which would permit us to assess if a consensus measurement obtained from a crowd of volunteers is sufficiently reproducible and accurate to be accredited as an AST device. Unfortunately, there is no international standard for Mycobacterial AST – the need to subject Mycobacteria to the same processes and standards as other bacteria has been argued elsewhere [@bib19] – we shall therefore tentatively apply the international standard for aerobic bacteria [@bib9] which requires the results of the proposed antibiotic susceptibility testing method to an appropriate reference method.\n",
"\n",
"Neither of the measurement methods used in constructing our reference consensus dataset has been endorsed, although broth microdilution using Middlebrook 7H9 media was recently selected by EUCAST as a reference method for determining _M. tuberculosis_ MICs [@bib19] but only for manually-prepared 96-well plates, permitting much larger numbers of wells for each drug. Nor has any software-based approach for reading MICs from 96-well microdilution plates been endorsed by EUCAST, the CLSI or any other international body. Despite this, and in lieu of any other reasonable approach, we shall treat the consensus MICs (the Expert +AMyGDA datset) as a reference dataset and apply ISO 20776–2 [@bib9].\n",
"\n",
"This requires a new AST method that measures MICs to be compared to the reference method using two key metrics: the _exact agreement_ and the _essential agreement_. The former is simply the proportion of definite readings which agree, whilst the latter allows for some variability and is defined as the “MIC result obtained by the AST device that is within plus or minus doubling dilution step from the MIC value established by the reference method” [@bib9]. To meet the standard any new AST method must be ≥95% reproducible and ≥90% accurate (both assessed using essential agreement) compared to the reference method [@bib9].\n",
"\n",
"## Variability in classifications\n",
"\n",
"Inevitably there is a large degree of variation in the classifications made by different volunteers of the same drug image. Examining all 112,163 classifications made by the volunteers on the 6205 drug images taken after 14 days incubation and comparing them to the consensus of the laboratory scientist and AMyGDA shows that a single volunteer is likely to exactly agree with the Expert +AMyGDA dataset 74.6% of the time, excluding cases where either concluded the drug image could not be read ([Supplementary file 1c](#supp1)). This rises to 86.4% when only considering essential agreement.\n",
"\n",
"The magnitude of agreement varies depending on the measured dilution: if the consensus view is that a drug image contains no growth, a single volunteer is likely to agree 64.1% of the time ([Figure 2A](#fig2)), however this falls to 47.8% if the consensus dataset indicates that the first four wells contain growth before rising to 94.5% when the laboratory scientist decides the dilution is 8. We hence recapitulate our earlier observation that drug images with no growth or growth in all wells are easier to read than drug images where only a subset of drug wells contain growth.\n",
"\n",
"figure: Figure 2.\n",
":::\n",
"\n",
"\n",
"### Heatmap showing how all the individual BashTheBug classifications (n=214,164) compare to the dilution measured by the laboratory scientist using the Thermo Fisher Vizion instrument after 14 days incubation (n=12,488).\n",
"\n",
"(**A**) The probability that a single volunteer exactly agrees with the Expert +AMyGDA dataset varies with the dilution. (**B**) The distribution of all dilutions in the Expert +AMyGDA dataset after 14 days incubation. The differences are due to different drugs having different numbers of wells as well as the varying levels of resistance in the supplied strains. NR includes both plates that could not be read due to issues with the control wells and problems with individual drugs such as skip wells. (**C**) The distribution of all dilutions measured by the BashTheBug volunteers. (**D**) A heatmap showing the concordance between the Expert +AMyGDA dataset and the classifications made by individual BashTheBug volunteers. Only cells with _gt_~0.1~% are labelled. (**E**) Two example drug images where both the Expert and AMyGDA assessed the MIC as being a dilution of 5 whilst a single volunteer decided no growth could be seen in the image. (**F**) Two example drug images where both the laboratory scientist and a volunteer agreed that the MIC was a dilution of 5. (**G**) Two example drug images where the laboratory scientist decided there was no growth in any of the wells, whilst a single volunteer decided there was growth in the first four wells.\n",
":::\n",
"{#fig2}\n",
"\n",
"figure: Figure 2—figure supplement 1.\n",
":::\n",
"\n",
"\n",
"### Heatmap showing how all the individual BashTheBug classifications (n=214,164) compare to the set of dilutions where the measurement made by the laboratory scientist using the Thermo Fisher Vizion instrument and a mirrored box after 14 days incubation concur (n=9402) (**A**).\n",
"\n",
"The probability that a single volunteer exactly agrees with the Expert dataset varies with the dilution. The distribution of all MIC dilutions after 14 days incubation read by (**B**) laboratory scientists and (**C**) BashTheBug volunteers. NR includes both plates that could not be read due to issues with the control wells and problems with individual drugs such as skip wells. (**D**) A heatmap showing how for each set of images assessed by the laboratory scientist has having a specific dilution as the MIC, the classifications made by BashTheBug volunteers varied considerably. It is normalised so that each row sums to 100% and only cells with >0.1 % are labelled.\n",
":::\n",
"{#fig2s1}\n",
"\n",
"The BashTheBug volunteers are likely to return a higher dilution than the Expert +AMyGDA consensus; this can be seen in the greater proportion of MICs with higher dilutions ([Figure 2B–D](#fig2)). For example a single volunteer is at least 5 ×, and often > 10 ×, more likely to return an MIC one greater than the reference rather than an MIC one lower than the reference ([Figure 2D](#fig2)). We shall return to this bias later.\n",
"\n",
"When the classification made by an individual volunteer does not agree with the consensus this is often (but not always) because they have misclassified the drug image ([Figure 2E–G](#fig2)). Comparing the classifications made by individual volunteers with the larger, but presumably less accurate, Expert dataset we see that an individual volunteer is less likely to agree with a single laboratory scientist with the overall level of exact agreement falling from 74.6% to 65.3% ([Supplementary file 1](#supp1), [Figure 2—figure supplement 1](#fig2s1)). Regardless of the comparison dataset used, it is clear that to achieve satisfactory levels of reproducibility and accuracy one must clearly ask _several_ volunteers to assess each drug image and then form a consensus measurement which can be compared to the reference measurement. How should we form that consensus?\n",
"\n",
"## Consensus\n",
"\n",
"There are a range of methods one can use to extract a consensus from a set of classifications; the simplest being majority voting, however, this is not practical since an outright majority is not guaranteed. Alternatively one may take the mode, mean or median of the classifications, although the former is not always defined and the last two do not always yield an integer. More sophisticated methods, such as the weighted-majority algorithm [@bib12], give weights to the classifiers based on their accuracy; however, this requires each volunteer to first classify a ground-truth dataset, which was not available at the start of the project. Given the high level of inequality in participation ([Figure 1C](#fig1)), such methods would be very difficult to apply in practice in our case. We shall therefore limit ourselves here to considering only the mean, median and mode. Since these methods all require the classifications to be numerical, we excluded all readings where the Expert +AMyGDA measurement and/or half or over of the volunteers decided the drug image could not be read. If the classification distribution was bi-modal, then the lower value of the dilution is returned. If necessary, the mean or median were also rounded down.\n",
"\n",
"## Reproducibility\n",
"\n",
"To create two consensus measurements by the volunteers of each drug image, two separate sets of 17 classifications were drawn with replacement. By applying the relevant method (mean, median or mode) a consensus dilution was arrived at for each set and then the two results compared. To begin with only drug images with 17 or more classifications were considered and this bootstrapping process was repeated ten times for each drug image in the Expert+AMyGDA dataset. Considering only those drug images taken after 14 days incubation ([Figure 3A](#fig3), [Supplementary file 1e](#supp1)), they are more likely to exactly agree with one another when the mode was applied (89.2 ± 0.1%) than the median (86.9 ± 0.1%) or mean (74.3 ± 0.1%). For the essential agreement, we find that the mean now performs best (97.2 ± 0.1%), followed by the median (94.2 ± 0.1%) and mode (94.1 ± 0.1%). Here and throughout, the standard error of the mean is quoted. As one might expect, the reproducibility for MICs measured by two laboratory scientists, as quantified by essential agreement, was previously shown to be slightly higher, at 97.9 % [@bib17].\n",
"\n",
"figure: Figure 3.\n",
":::\n",
"\n",
"\n",
"### Taking the mean of 17 classifications is ≥95% reproducible whilst applying either the median or mode is ≥90% accurate.\n",
"\n",
"(**A**) Only calculating the mean of 17 classifications achieves an essential agreement ≥95% for reproducibility [@bib9], followed by the median and the mode. (**B**) Heatmaps of the consensus formed via the mean, median or mode after 14 days incubation. Only drug images from the Expert + AMyGDA dataset are included. (**C**) The essential agreement between a consensus dilution formed from 17 classifications using the median or mode and the consensus Expert +AMyGDA dilution both exceed the required 90% threshold [@bib9]. (**D**) The heatmaps clearly show how the volunteer consensus dilution is likely to be the same or greater than the Expert + AMyGDA consensus.\n",
":::\n",
"{#fig3}\n",
"\n",
"figure: Figure 3—figure supplement 1.\n",
":::\n",
"\n",
"\n",
"### Taking the mean of 17 classifications is ≥95% reproducible whilst none of the methods reach have an essential agreement for accuracy of ≥90% when using the Expert dataset.\n",
"\n",
"(**A**) Only calculating the mean of 17 classifications achieves an essential agreement ≥95% for reproducibility [@bib9], followed by the median and then the mode. There is no specified threshold for exact agreement; the trend is reversed with the mode performing best, followed by the median and then the mean. (**B**) Heatmaps of the consensus formed via the mean, median, or mode after 14 days incubation. Each consensus dilution is a different selection, with replacement, of the original classifications. Drug images from the larger Expert dataset are included. (**C**) The essential agreement between a consensus dilution formed from 17 classifications using the median or mode and the consensus Expert dilution is ≥ 90%, which is the required threshold [@bib9]. (**D**) The heatmaps clearly show how the volunteer consensus dilution is likely to be the same or greater than the Expert consensus.\n",
":::\n",
"{#fig3s1}\n",
"\n",
"Hence only the mean exceeds the threshold for reproducibility [@bib9] when 17 classifications are used to build a consensus. Repeating the analysis for the drug images in the larger Expert dataset yields the same conclusion ([Figure 3—figure supplement 1](#fig3s1)). The heatmaps ([Figure 3B](#fig3)) show how two consensus measurements arrived at via the mean tend to be similar but not necessarily identical to one another, whilst two consensus measurements derived using the mode are more likely to agree with one another but also are more likely to arrive at very different values. The median sits in between these two extremes.\n",
"\n",
"## Accuracy\n",
"\n",
"Comparing the consensus measurements from the volunteers to the set of MICs in the Expert+AMyGDA dataset yields a different picture ([Figure 3C](#fig3)). The mode exactly agrees with the reference 80.9 ± 0.1% of the time, followed by the median (78.1 ± 0.1%) and then mean (68.4 ± 0.1%). The mean, despite performing best for reproducibility, has the lowest level of essential agreement (as well as exact agreement) with the Expert+AMyGDA readings (88.5 ± 0.1%), with the median (90.2 ± 0.1%) and mode (91.0 ± 0.1%) both exceeding the 90% accuracy threshold [@bib9].\n",
"\n",
"The heatmaps show how the consensus dilution of the classifications made by the volunteers is much more likely to be higher than the Expert+AMyGDA measurement than lower ([Figure 3D](#fig3)), regardless of the consensus method, indicating perhaps that volunteers are more likely than laboratory scientists to classify any dark regions in a well as bacterial growth, or that laboratory scientists are more willing to discount some features as artefacts for example air bubbles or sediment. Repeating the analysis using the Expert dataset ([Figure 3—figure supplement 1](#fig3s1)) leads to lower values for the exact and essential agreements for all consensus methods – this is to be expected since this the Expert reference dataset contains a larger proportion of errors than the Expert+AMyGDA dataset.\n",
"\n",
"## Which method to choose?\n",
"\n",
"Despite being the most reproducible method as measured by essential agreement, we discount the mean since it suffers from relatively poor levels of exact agreement for both reproducibility and accuracy and its performance falls faster than the other methods when $n$ is decreased. The median and mode have very similar reproducibilities and accuracy and we conclude they perform equally well. We can infer from this that bi-modal classification distributions are rare and that the median is often identical to the mode.\n",
"\n",
"## Reducing the number of classifications\n",
"\n",
"Clearly it would be desirable and ethical to only require the volunteers to complete the minimum number of classifications to achieve an acceptable result. The simplest way to do this is to decrease the number of classifications, $n$, before a drug image is retired – this reduces both the reproducibility and accuracy of the consensus measurements ([Figure 4](#fig4), [Supplementary file 1e, f](#supp1)), however perhaps not by as much as one might expect. The mean exceeds the essential agreement ≥ 95% reproducibility threshold for $n\\ge 13$, whilst the mode and the median satisfy the accuracy criterion of essential agreement ≥ 90% for $n\\ge 3$ and $n\\ge 11$, respectively ([Figure 4](#fig4), [Supplementary file 1f](#supp1)). Similar trends are observed when the Expert dataset is used as the reference ([Figure 4—figure supplement 1](#fig4s1)) or when the plates are incubated for different periods of time ([Figure 4—figure supplements 2](#fig4s2) and [!number(3)](#fig4s3)). Accuracy is hence less sensitive than reproducibility to reducing the number of classifications used to build the consensus and, depending on the consensus method used, the number of classifications can be reduced whilst still maintaining acceptable levels of accuracy.\n",
"\n",
"figure: Figure 4.\n",
":::\n",
"\n",
"\n",
"### Reducing the number of classifications, $n$, used to build the consensus dilution decreases the reproducibility and accuracy of the consensus measurement.\n",
"\n",
"(**A**) The consensus dilution becomes less reproducible as the number of classifications is reduced, as measured by both the exact and essential agreements. (**B**) Likewise, the consensus dilution becomes less accurate as the number of classifications is decreased, however the highest level of exact agreement using the mean is obtained when $n=3$ and the mode, and to a lesser extent the median, are relatively insensitive to the number of classifications. These data are all with respect to the Expert +AMyGDA dataset.\n",
":::\n",
"{#fig4}\n",
"\n",
"figure: Figure 4—figure supplement 1.\n",
":::\n",
"\n",
"\n",
"### Reducing the number of classifications, $n$, used to build the consensus dilution decreases the reproducibility and accuracy of the consensus measurement.\n",
"\n",
"(**A**) The consensus dilution becomes less reproducible as the number of classifications is reduced, as measured by both the exact and essential agreements. (**B**) Likewise, the consensus dilution becomes less accurate as the number of classifications is decreased, however the highest level of exact agreement using the mean is obtained when $n=3$ and the mode, and to a lesser extent the median, are relatively insensitive to the number of classifications. These data are all with respect to the Expert dataset.\n",
":::\n",
"{#fig4s1}\n",
"\n",
"figure: Figure 4—figure supplement 2.\n",
":::\n",
"\n",
"\n",
"### Altering the number of days incubation does not markedly affect the observed trends in reproducibility.\n",
"\n",
"Shown are results for the Expert +AMyGDA dataset after (**A**) 7, (**B**) 10, (**C**) 14 and (**D**) 21 days of incubation. A previous study [@bib17] showed that optimal results were achieved after 14 days incubation.\n",
":::\n",
"{#fig4s2}\n",
"\n",
"figure: Figure 4—figure supplement 3.\n",
":::\n",
"\n",
"\n",
"### Altering the number of days incubation does not markedly affect the observed trends in accuracy.\n",
"\n",
"Shown are results for the Expert +AMyGDA dataset after (**A**) 7, (**B**) 10, (**C**) 14 and (**D**) 21 days of incubation. A previous study [@bib17] showed that optimal results were achieved after 14 days incubation.\n",
":::\n",
"{#fig4s3}\n",
"\n",
"figure: Figure 4—figure supplement 4.\n",
":::\n",
"\n",
"\n",
"### Segmenting the drug images by the mean amount of growth in the positive control wells ([Figure 6—figure supplement 3](#fig6s3)) does not markedly affect the reproducibility of the three consensus methods.\n",
"\n",
"The plates are split into those with (**A**) low (≤ 10 %) growth, (**B**) medium (10 < growth ≤) growth and (**C**) high (> 50 %) growth. The drug images from the Expert +AMyGDA dataset were used and the proportion with MIC is the proportion of consensus readings that are a definite numerical minimum inhibitory concentration.\n",
":::\n",
"{#fig4s4}\n",
"\n",
"figure: Figure 4—figure supplement 5.\n",
":::\n",
"\n",
"\n",
"### Segmenting the drug images by the mean amount of growth in the positive control wells ([Figure 6—figure supplement 3](#fig6s3)) does not markedly affect the accuracy of the three consensus methods.\n",
"\n",
"The plates are split into those with (**A**) low (≤ 10% %) growth, (**B**) medium (10 < growth ≤ 50 %) growth and (**C**) high (> 50 %) growth. The drug images from the Expert +AMyGDA dataset were used and the proportion with MIC is the proportion of consensus readings that are a definite numerical minimum inhibitory concentration.\n",
":::\n",
"{#fig4s5}\n",
"\n",
"## Can we improve matters?\n",
"\n",
"Retiring all drug images after a fixed number of classifications is simple but does not take account of the relative difficulty of the classification task. If one was able to group the drug images by difficulty, either before upload or dynamically during classification, then one could optimally target the work undertaken by the volunteers. Due to the inherent difficulties in culturing _M. tuberculosis_, there is a broad distribution of growth in the positive control wells after 14 days incubation ([Figure 6—figure supplement 3](#fig6s3)). One might expect that drug images with poor growth would be more challenging to classify, however, segmenting by low, medium and high growth shows the amount of growth in the positive control wells has little effect on either the reproducibility ([Figure 4—figure supplement 4](#fig4s4)) or accuracy ([Figure 4—figure supplement 5](#fig4s5)), regardless of the consensus method and number of classifications employed.\n",
"\n",
"Alternatively one could use the first few classifications performed by the volunteers to assess the difficulty of each drug image. For example, if the first $n$ volunteers all return the same classification, then it is reasonable to assume that this is a straightforward image and it can be retired, with the remainder accruing additional classifications. Ideally one would want to develop an algorithm that assessed the likelihood of the classification not being altered by more classifications to allow a dynamic decision about when to halt, however, applying such an approach was not possible at the time in the Zooniverse.\n",
"\n",
"To estimate the potential value in applying a simple approach to dynamically retiring drug images, we shall consider applying the median after 14 days of incubation and will arbitrarily retire a drug image if the first three volunteers all made the same classification, with all other drug images being retired after 17 classifications. This simple protocol reduces the number of classifications required to $n=8.8$, a reduction of 48%, and the reproducibility, as measured by exact agreement, rises from 86.8% to 87.6%, whilst the essential agreement remains unchanged (94.2–94.4%). The accuracy, assessed in the same way, behaves similarly with the exact agreement increasing from 78.1% to 78.8% with the essential agreement remaining unaltered (90.2–90.3%). Hence retiring some of the drug images at $n=3$ not only dramatically reduces the number of classifications required but also improves the result in a few cases, presumably because the subsequent classifications have a small chance of altering the dilution by a single unit, hence worsening the exact agreement but not affecting the essential agreement.\n",
"\n",
"A fairer test is to ask if this dynamic approach improves performance if we are constrained to a fixed number of total classifications: if we choose $n=9$, then the reproducibility of the median (as measured by exact and essential agreements) improves from 83.0–92.2% to 87.6–94.4% and the accuracy, measured in the same way, improves slightly from 77.2–89.6% to 78.8–90.3%. We therefore conclude that even a simple dynamic approach to retiring images would minimise the work done by the volunteers / allow more images to be classified.\n",
"\n",
"## Variation by drug\n",
"\n",
"So far we have analysed the reproducibility and accuracy of consensus MICs obtained from a crowd of volunteers, thereby aggregating the results for each of the 13 anti-tuberculars (excl. PAS) present on the UKMYC5 plate design. The reproducibility of each drug, as measured by the exact and essential agreements, varies between 79.6–91.4% and 90.8–97.0%, respectively ([Figure 5A](#fig5)). This is consistent with an earlier study which found that intra- and inter-laboratory reproducibility when the drugs were read by a scientist using a Vizion instrument varied between 93.4-99.3 % and 87.1-99.1 % (as measured by essential agreement, excl. PAS), respectively [@bib17]. Earlier we showed that the reproducibility of the whole plate under these conditions when assessed using the essential agreement is 94.2 ± 0.1% ([Figure 3A](#fig3)) – this is below the 95% threshold specified by an international standard for aerobic bacteria [@bib9]. Applying the same threshold to each drug we find that five out of the 13 drugs meet or exceed the threshold whilst the plate as a whole does not. The accuracy of each drug varies more widely: between 48.4–88.9% and 82.8–94.7% when assessed using the exact and essential agreement, respectively. Hence whilst the accuracy of the plate as a whole was 90.2 ± 0.1%, just exceeding the 90% threshold, only six out of 13 drugs surpassed the same threshold.\n",
"\n",
"figure: Figure 5.\n",
":::\n",
"\n",
"\n",
"### The reproducibility and accuracy of the consensus MICs varies by drug.\n",
"\n",
"Consensus MICs were arrived at by taking the median of 17 classifications after 14 days incubation. The essential and exact agreements are drawn as red and green bars, respectively. For the former the minimum thresholds required are 95% and 90% for the reproducibility and accuracy, respectively [@bib9]. See ([Figure 5—figure supplement 1](#fig5s1)) for the other consensus methods.\n",
":::\n",
"{#fig5}\n",
"\n",
"figure: Figure 5—figure supplement 1.\n",
":::\n",
"\n",
"\n",
"### The reproducibility and accuracy after 14 days incubation of the 13 antibiotics on the UKMYC5 plate.\n",
"\n",
"A total of 17 classifications were used for each measurement and either the mean or mode was used to obtain a consensus reading of the (**A**) reproducibility and (**B**) accuracy. The essential agreement is drawn in red and the required thresholds are 95% and 90% for the reproducibility and accuracy, respectively [@bib9]. The exact agreement is drawn in green and no threshold is defined. The drug abbreviations are defined in ([Figure 6—figure supplement 1](#fig6s1)). The dataset used was Expert +AMyGDA.\n",
":::\n",
"{#fig5s1}\n",
"\n",
"The variation in reproducibility and accuracy between anti-tuberculars, as well as between the exact and essential agreement for a single compound, is due to a number of factors, not limited to the number and concentration range of the wells on the plate design, the mechanisms of both action and resistance, the prevalence of resistance and the degradation rate after lyophilisation, both during storage and after inoculation. For example, kanamycin passes the reproducibility and accuracy thresholds we have adopted and this is likely due to there being relatively few (five) drug wells on the UKMYC5 plate ([Figure 6—figure supplement 1](#fig6s1)) and the mechanism of resistance being such a substantial proportion of samples either do not grow in any well, or grow in all the drug wells, making measurement more straightforward. The mechanism of action of each compound is likely to affect how ‘easy’ it is to determine the minimum inhibitory concentration. From the striking differences in exact and essential accuracies of reading ethambutol, moxifloxacin and linezolid we hypothesise the Mycobacterial growth diminishes more gradually with increasing concentration for these drugs, rather than coming to an abrupt end, as it does for other compounds.\n",
"\n",
"This whole analysis could be considered somewhat moot since the UKMYC5 96-well plate would be treated as a single entity (or medical device) if accreditation were to be sought and therefore the results for individual compounds would likely not be considered. One unintended consequence of the current standards is therefore that one could improve the performance of a plate design by dropping compounds with lower-than-average performance, even if this is clinically not desirable, rather than work to for example improve the performance of the measurement methods.\n",
"\n",
"# Discussion\n",
"\n",
"A crowd of volunteers can reproducibly and accurately measure the growth of a bacterial pathogen on a 96-well broth microdilution plate, thereby demonstrating the potential for clinical microbiology to embrace and combine contrasting measurement methods. No Mycobacterial antibiotic susceptibility testing standard exists, although efforts are underway to establish a reference method [@bib18], and so we applied the standard for antibiotic susceptibility testing of aerobic bacteria [@bib9]. Forming a consensus by applying the mode or median to 17 independent classifications performs better overall than the mean, and the reproducibility of both these methods, as measured by the essential agreement, is 94.2% and 94.1%, respectively ([Figure 3](#fig3)). This is below than the 95% threshold set by ISO for aerobic bacteria and therefore the volunteers do not pass this standard. The accuracy of the crowd, as measured by the essential agreement, is 90.2% and 91.0% when the median and mode, respectively, are applied to produce a consensus measurement – these values are above the required 90% threshold [@bib9], and therefore the volunteers are sufficiently accurate (but not quite reproducible enough) to be classified as an antibiotic susceptibility testing (AST) device.\n",
"\n",
"The BashTheBug project has therefore been successful in its primary aim, i.e. the consensus readings made by the crowd of volunteers have been successfully used within the CRyPTIC project to reduce the measurement error in the dataset of 268,281 (13 drugs × 20,637 samples) minimum inhibitory concentrations. CRyPTIC has already significantly contributed to the first resistance catalogue published by the World Health Organization [@bib26], carried out the largest genome-wide association study of _M. tuberculosis_ [@bib22] and assessed the magnitude of MIC-shifts for the most common mutations that confer resistance [@bib22]. The intention is to accelerate the shift from culture-based to genetics-based AST, which is already faster and cheaper than the culture-based alternatives for tuberculosis [@bib16] and this is well underway [@bib25]. Ultimately shifting to a genetics-based paradigm, where the susceptibility of a pathogen to an antibiotic is inferred from the genome of the pathogen, potentially offers a route to making pathogen diagnostics much more widely available. On a more personal note, it has been very fulfilling engaging with the public through the BashTheBug project and we urge other scientists to consider citizen science if it is appropriate for their research. It is our experience that, whilst the sheer volume of classifications a crowd can achieve can be staggering, it requires us, as researchers, to invest time and effort in regularly communicating with the volunteers which we mainly achieved through our blog, Twitter account, the Zooniverse talk boards and newsletter emails.\n",
"\n",
"The volunteers are fast, taking on average 3.5 s per drug image, and therefore a single plate requires slightly less than 13 min of volunteer time to read if 17 classifications are amassed for each drug. Reducing the number of classifications before an image is retired reduces the reproducibility and accuracy, but not by as much as one might expect. A more nuanced approach would be to retire a drug image early if the first few classifications are identical, however it was not possible to define this type of dynamical rule in the Zooniverse portal when these data were collected. The level of participation by the volunteers was very unequal with a small cadre of volunteers doing very large numbers with ten volunteers doing, on average, over 10,000 classifications each which is more than many of the laboratory scientists who are considered the experts!\n",
"\n",
"Compared to the measurements taken by the laboratory scientists, the consensus dilution arrived at by the volunteers tends to be higher, indicating a bias to overcall ([Figure 3—figure supplement 1](#fig3s1)), which is supported by anecdotal observations of people classifying drug images at public engagement events where they often choose a higher dilution ‘to be on the safe side’. As volunteers do more classifications they become more experienced and, on the whole, the magnitude of the bias reduces but does not disappear ([Figure 6—figure supplement 5](#fig6s5)). By contrast, the AMyGDA computer software has been noted to have the opposite bias – that is is more likely to undercall compared to the expert [@bib6]. At present in the CRyPTIC project these biases are ignored, that is if the AMyGDA MIC and the consensus reached by BashTheBug agree it is assumed that is a correct measurement. A more nuanced approach that took into account these systematic biases would, we expect, reduce the levels of measurement error in the CRyPTIC MIC dataset yet further and slightly increase the number of MICs where two or more measurement methods agree with one another [@bib2].\n",
"\n",
"The reproducibility and accuracy of any method used to read a 96-well microtitre plate, whether that is laboratory scientists using a Thermo Fisher Vizion instrument or citizen scientists visually examining drug images, depends on a range of factors from the prevalence of drug-resistant samples in the dataset to which drugs are included in the plate design and the number and concentrations of their allotted wells. For the UKMYC5 plate design, both the Expert and either the AMyGDA or BashTheBug measurements are more likely to agree with one another at low dilutions where there is little or no _M. tuberculosis_ growth in the drug wells ([Supplementary file 1d, i](#supp1)), hence reducing the number of resistant samples would artificially ‘improve’ performance yet the standards do not specify the degree of resistance in any test dataset [@bib9]. The requirement to have quality control strains that have definite growth in all the drug wells unintentionally mitigates against this risk. The dataset used here was based on 19 external quality assessment strains and therefore whilst it included some degree of resistance for all 13 drugs, there was only a single strain resistant to clofazimine, bedaquiline or delamanid and no strain was resistant to linezolid [@bib17]. For isoniazid and rifampicin, eight and seven of the 19 EQA strains, respectively were resistant and hence the prevalence of resistance for these drugs is much greater than would be expected to be encountered in most countries. Clearly studies including a much more diverse range of strains, for example clinical isolates, would be more definitive. Since the 13 antituberculars on the UKMYC5 plate ([Figure 6—figure supplement 1](#fig6s1)) also all perform differently ([Figure 5](#fig5)) different plate designs will perform differently, which is important as it is the plate that would be accredited, rather than the individual compounds.\n",
"\n",
"Although the primary aim of this study was to assess whether the measurements produced by a crowd of volunteers are sufficiently reproducible and accurate to help reduce the measurement error in datasets containing large numbers of microtitre plates (as is being collected by the CRyPTIC project and others) the resulting dataset of classifications is ideally suited to train machine-learning models. This is increasingly recognised as an important use of citizen science [@bib24] and one could envisage training a light-weight machine-learning algorithm able to run on a mobile device which, by taking a photograph of a 96-well plate, could automatically read the minimum inhibitory concentrations. The best use of such a device would likely be to act as a double check for readings taken by the laboratory scientist. Alternatively, one could build a hybrid approach where e.g. small crowds of experts could examine plates used in a clinical microbiology service – these could be particularly difficult drug images or could be a random sample for quality assurance purposes. This type of hybrid approach would also help with training laboratory scientists which would help reduce the barrier to using 96-well microtitre plates for _M. tuberculosis_ AST in clinical microbiology laboratories, especially in low- and middle-income countries. Finally, it is likely that each volunteer has their own individual bias and variability and constructing consensus methods [@bib28] that take these into account would likely further improve the performance of crowds of citizen scientists.\n",
"\n",
"# Materials and methods\n",
"\n",
"## Plate design\n",
"\n",
"The CRyPTIC project has collected a large number (20,637) of clinical TB samples, each having the MICs of a panel of 13 antibiotics measured using a 96-well broth microdilution plate with the majority of samples also undergoing whole genome sequencing. Samples were collected by 14 laboratories based in 11 countries spread across five continents and included all the major lineages and were biased towards drug resistance [@bib2; @bib23]. Two plate designs were used, with the second (UKMYC6) being an evolution of the first (UKMYC5) plate design. In this paper we shall focus on data collected using the UKMYC5 plate design which is itself a variant of the MYCOTB 96-well microdilution plate manufactured by Thermo Fisher containing 14 anti-TB drugs. UKMYC5 includes two repurposed compounds (linezolid and clofazimine) and two new compounds (delamanid and bedaquiline). Since 96-well plates have 8 rows and 12 columns, fitting 14 drugs, alongside two positive control wells, onto the plate necessitated a complex design ([Figure 6—figure supplement 1](#fig6s1)).\n",
"\n",
"Each of the antibiotics on the UKMYC5 plate is present at 5–8 concentrations, each double the previous. The smallest concentration that prevents growth of _M. tuberculosis_ after two weeks of incubation is the minimum inhibitory concentration (MIC). Since _M. tuberculosis_ is notoriously difficult to culture and inspect, relying on a reading taken by a single expert (laboratory scientist) would likely have led to high levels of errors in the dataset.\n",
"\n",
"## Image dataset\n",
"\n",
"The images considered by the crowd of volunteers originated from an earlier study whose aim was to assess if the UKMYC5 plate was sufficiently reproducible and accurate for use as a high-throughput research MIC measurement device [@bib17]. Seven CRyPTIC laboratories were each sent thirty one vials containing one of 19 external quality assessment (EQA) _M. tuberculosis_ strains, including the reference strain H37Rv ATCC 27294 [@bib11]. The EQA strains varied in their resistance profiles [@bib17]. Thirty of the vials were blinded and the last contained the reference strain. Within the thirty vials, some of EQA strains were duplicated and two of the blinded strains were also the reference strain. Since some labs only received a subset of the 31 vials ([Supplementary file 1a](#supp1)), a total of 447 plates were inoculated and then incubated for 3 weeks ([Figure 6A](#fig6)). Due to the slow growth rate of _M. tuberculosis_, minimum inhibitory concentrations of the 14 drugs on the plate were measured after 7, 10, 14, and 21 days by two scientists using a Thermo Fisher Sensititre Vizion Digital MIC viewing system, a mirrored-box and a microscope. One or two photographs were also taken each time using the Vizion instrument ([Figure 6B](#fig6)).\n",
"\n",
"figure: Figure 6.\n",
":::\n",
"\n",
"\n",
"### Each UKMYC5 plate was read by an Expert, by some software (AMyGDA) and by at least 17 citizen scientist volunteers via the BashTheBug project.\n",
"\n",
"(**A**) 447 UKMYC5 plates were prepared and read after 7, 10, 14 and 21 days incubation. (**B**) The minimum inhibitory concentrations (MIC) for the 14 drugs on each plate were read by an by Expert, using a Vizion instrument. The Vizion also took a photograph which was subsequently analysed by AMyGDA – this software then composited 14 drug images from each photograph, each containing an image of the two positive control wells. To allow data from different drugs to be aggregated, all MICs were converted to dilutions. (**C**) All drug images were then uploaded to the Zooniverse platform before being shown to volunteers through their web browser. Images were retired once they had been classified by 17 different volunteers. Classification data were downloaded and processed using two Python modules (pyniverse +bashthebug) before consensus measurements being built using different methods.\n",
":::\n",
"{#fig6}\n",
"\n",
"figure: Figure 6—figure supplement 1.\n",
":::\n",
"\n",
"\n",
"### The UKMYC5 plate contains 14 different anti-TB drugs.\n",
"\n",
"A previous study [@bib17] showed that _para_-aminosalicylic acid (PAS) performed poorly and it has been removed from the subsequent UKMYC6 plate design. We have therefore excluded this drug from all analyses. Each drug was contained in 5, 6, 7, or 8 wells with each well having double the concentration of drug as the one before. The concentration of the first and last well in each drug series is labelled (mg/L). Two wells contain no drug and are therefore positive control wells.\n",
":::\n",
"{#fig6s1}\n",
"\n",
"figure: Figure 6—figure supplement 2.\n",
":::\n",
"\n",
"\n",
"### Although the retirement limit within the Zooniverse platform was set to 17, over 1800 images received more classifications than this and a small number were only classified 15 or 16 times.\n",
":::\n",
"{#fig6s2}\n",
"\n",
"figure: Figure 6—figure supplement 3.\n",
":::\n",
"\n",
"\n",
"### The Expert +AMyGDA consensus dataset has the same distribution of bacterial growth in the positive control wells as the Expert dataset after 14 days incubation.\n",
"\n",
"(**A**) The distribution of the mean positive control well growth, as measured by AMyGDA, for the Expert +AMyGDA dataset. The dataset is arbitrarily split into three categories: low (<10%), medium (10 ≤ growth < 50 %) and high (≥ 50 %) growth. The proportions of the dataset in each category are labelled. (**B**) The distribution of the mean positive control well growth, as measured by AMyGDA, for the Expert dataset. There are around twice as many plates in this dataset ([Supplementary file 1c](#supp1)).\n",
":::\n",
"{#fig6s3}\n",
"\n",
"figure: Figure 6—figure supplement 4.\n",
":::\n",
"\n",
"\n",
"### The Expert +AMyGDA dataset has a greater proportion of drug images with low dilutions compared to the Expert dataset.\n",
"\n",
"The growth of the bacteria is also evident as the number of days the sample was incubated for is increased.\n",
":::\n",
"{#fig6s4}\n",
"\n",
"figure: Figure 6—figure supplement 5.\n",
":::\n",
"\n",
"\n",
"### The average bias per volunteer decreases with experience.\n",
"\n",
"The average bias per volunteer, as defined by the difference between a volunteer’s reading and that from the Expert +AMyGDA dataset, is plotted against the total number of classifications done by each volunteer. Only volunteers who have done 10 or more classifications are plotted.\n",
":::\n",
"{#fig6s5}\n",
"\n",
"The study showed that the UKMYC5 plate is reproducible and that it is optimal to read the plate using either a Thermo Fisher Vizion instrument or a mirrored-box after 14 days of incubation [@bib17]. It also showed that _para_-aminosalicylic acid (PAS) was not reproducible on the plate and therefore this drug is excluded in all subsequent analyses. Each image was processed and analysed by some bespoke software, the Automated Mycobacterial Growth Detection Algorithm (AMyGDA), that segmented each photograph, thereby providing a second independent MIC reading of all the drugs on each plate ([Figure 6B](#fig6); [@bib6]).\n",
"\n",
"Early internal tests using the Zooniverse platform showed that asking a volunteer to examine all 96 wells on a plate was too arduous a task. We therefore observed a clinical microbiologist as she examined several photographs of UKMYC5 plates. Rather than considering the _absolute_ growth in each well, she was constantly comparing the growth in the wells containing antibiotic back to the positive control wells and therefore was judging what constituted growth _relative_ to how well the isolate had grown in the absence of drug. A suitable task is therefore to classify the growth in the wells for a single drug as long as the positive control wells are also provided. The AMyGDA software was therefore modified to composite such _drug images_ ([Figure 6B](#fig6)).\n",
"\n",
"Each UKMYC5 plate yielded 14 composite images, one for each drug. Throughout the following analysis we shall aggregate all the data from the different drugs on the UKMYC5 plate. To facilitate this we shall therefore consider the _dilution_, which is defined as the number of the well in the drug image with the lowest antibiotic concentration which prevents bacterial growth, rather than the minimum inhibitory concentration. Following upload to the Zooniverse platform, the retirement threshold was set to 17 classifications, however some images attracted additional classifications with 421 images having ≥ 34 classifications whilst 89 have 100 or greater ([Supplementary file 1b](#supp1), [Figure 6—figure supplement 2](#fig6s2)). This is due, we believe, to images not being removed when retired, but instead having a ”Finished!” sticker added and hence images can accrue additional classifications, especially at times when the volunteers have completed all the uploaded images. Each volunteer is only permitted to classify each image once.\n",
"\n",
"## Analysis\n",
"\n",
"The resulting classifications were downloaded from the Zooniverse platform, either by a web browser or using the panoptes-cli command line tool [@bib14]. Two Python modules were written to parse, store, manipulate and graph this classification data. The first, pyniverse [@bib4], is designed to be generic for Zooniverse projects whilst the second, bashthebug [@bib5], inherits from the first and adds functionality specific to BashTheBug ([Figure 6C](#fig6)). Both are freely available to download and use. These Python modules output several Pandas [@bib13] dataframes which were then indexed, filtered and joined to other dataframes containing the sample information and the MIC readings taken by the expert and AMyGDA software. AMyGDA also measured the growth in the two positive control wells and this was also recorded in a dataframe. All subsequent analysis was performed using Python3 in a jupyter-notebook (see Data Availability Statement) and all graphs were plotted using matplotlib.\n",
"\n",
"## Engagement\n",
"\n",
"In addition to the Zooniverse project page, which contained background information, a tutorial and FAQs, we setup a blog [@bib3] and a variety of social media channels, focussing mainly on Twitter (@bashthebug). These all used a professionally designed logo and typeface ([Figure 6C](#fig6)), allowing instantaneous recognition of the project, which is important since the Zooniverse platform hosts tens of projects at any one time, and to indirectly convey that this is a professional project and therefore of scientific and societal importance. Since the blog was launched in March 2017 we have written 71 posts, attracting 7,393 visitors who made 13,811 views. At the time of writing, the Twitter account, @bashthebug, has 393 followers and has tweeted 400 times. Finally, the volunteers interacted with one another as well as the project team via the BashTheBug Talk Boards on the Zooniverse platform. A total of 6,255 posts were made by 1042 individuals on 4834 topics. During the course of the project, one of our more experienced volunteers (EMLB) became a de facto moderator by answering so many of the questions posted (>500) which we recognised by giving her moderator status and she is also an author of this paper."
]
}
]
}