<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD with MathML3 v1.2 20190208//EN" "JATS-archivearticle1-mathml3.dtd"> <article xmlns:ali="http://www.niso.org/schemas/ali/1.0/" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" article-type="research-article" dtd-version="1.2"><front><journal-meta><journal-id journal-id-type="nlm-ta">elife</journal-id><journal-id journal-id-type="publisher-id">eLife</journal-id><journal-title-group><journal-title>eLife</journal-title></journal-title-group><issn publication-format="electronic" pub-type="epub">2050-084X</issn><publisher><publisher-name>eLife Sciences Publications, Ltd</publisher-name></publisher></journal-meta><article-meta><article-id pub-id-type="publisher-id">75046</article-id><article-id pub-id-type="doi">10.7554/eLife.75046</article-id><article-categories><subj-group subj-group-type="display-channel"><subject>Research Article</subject></subj-group><subj-group subj-group-type="heading"><subject>Microbiology and Infectious Disease</subject></subj-group></article-categories><title-group><article-title>A crowd of BashTheBug volunteers reproducibly and accurately measure the minimum inhibitory concentrations of 13 antitubercular drugs from photographs of 96-well broth microdilution plates</article-title></title-group><contrib-group><contrib contrib-type="author" corresp="yes" id="author-37155"><name><surname>Fowler</surname><given-names>Philip W</given-names></name><contrib-id authenticated="true" contrib-id-type="orcid">https://orcid.org/0000-0003-0912-4483</contrib-id><email>philip.fowler@ndm.ox.ac.uk</email><xref ref-type="aff" rid="aff1">1</xref><xref ref-type="other" rid="fund1"/><xref ref-type="other" rid="fund2"/><xref ref-type="other" rid="fund3"/><xref ref-type="fn" rid="con1"/><xref ref-type="fn" rid="conf1"/></contrib><contrib contrib-type="author" id="author-259847"><name><surname>Wright</surname><given-names>Carla</given-names></name><xref ref-type="aff" rid="aff1">1</xref><xref ref-type="other" rid="fund1"/><xref ref-type="other" rid="fund2"/><xref ref-type="fn" rid="con2"/><xref ref-type="fn" rid="conf2"/></contrib><contrib contrib-type="author" id="author-207315"><name><surname>Spiers</surname><given-names>Helen</given-names></name><xref ref-type="aff" rid="aff2">2</xref><xref ref-type="aff" rid="aff3">3</xref><xref ref-type="fn" rid="con3"/><xref ref-type="fn" rid="conf2"/></contrib><contrib contrib-type="author" id="author-259848"><name><surname>Zhu</surname><given-names>Tingting</given-names></name><xref ref-type="aff" rid="aff4">4</xref><xref ref-type="fn" rid="con4"/><xref ref-type="fn" rid="conf2"/></contrib><contrib contrib-type="author" id="author-260010"><name><surname>Baeten</surname><given-names>Elisabeth ML</given-names></name><xref ref-type="aff" rid="aff5">5</xref><xref ref-type="fn" rid="con5"/><xref ref-type="fn" rid="conf2"/></contrib><contrib contrib-type="author" id="author-196178"><name><surname>Hoosdally</surname><given-names>Sarah W</given-names></name><xref ref-type="aff" rid="aff1">1</xref><xref ref-type="other" rid="fund1"/><xref ref-type="other" rid="fund2"/><xref ref-type="fn" rid="con6"/><xref ref-type="fn" rid="conf2"/></contrib><contrib contrib-type="author" id="author-259849"><name><surname>Gibertoni Cruz</surname><given-names>Ana L</given-names></name><contrib-id authenticated="true" contrib-id-type="orcid">https://orcid.org/0000-0002-9473-2215</contrib-id><xref ref-type="aff" rid="aff1">1</xref><xref ref-type="other" rid="fund1"/><xref ref-type="other" rid="fund2"/><xref ref-type="fn" rid="con7"/><xref ref-type="fn" rid="conf2"/></contrib><contrib contrib-type="author" id="author-259851"><name><surname>Roohi</surname><given-names>Aysha</given-names></name><xref ref-type="aff" rid="aff1">1</xref><xref ref-type="other" rid="fund1"/><xref ref-type="other" rid="fund2"/><xref ref-type="fn" rid="con8"/><xref ref-type="fn" rid="conf2"/></contrib><contrib contrib-type="author" id="author-259850"><name><surname>Kouchaki</surname><given-names>Samaneh</given-names></name><xref ref-type="aff" rid="aff4">4</xref><xref ref-type="other" rid="fund1"/><xref ref-type="other" rid="fund2"/><xref ref-type="fn" rid="con9"/><xref ref-type="fn" rid="conf2"/></contrib><contrib contrib-type="author" id="author-196186"><name><surname>Walker</surname><given-names>Timothy M</given-names></name><contrib-id authenticated="true" contrib-id-type="orcid">https://orcid.org/0000-0003-0421-9264</contrib-id><xref ref-type="aff" rid="aff1">1</xref><xref ref-type="other" rid="fund1"/><xref ref-type="other" rid="fund2"/><xref ref-type="fn" rid="con10"/><xref ref-type="fn" rid="conf2"/></contrib><contrib contrib-type="author" id="author-94983"><name><surname>Peto</surname><given-names>Timothy EA</given-names></name><xref ref-type="aff" rid="aff1">1</xref><xref ref-type="other" rid="fund1"/><xref ref-type="other" rid="fund2"/><xref ref-type="fn" rid="con11"/><xref ref-type="fn" rid="conf2"/></contrib><contrib contrib-type="author" id="author-259852"><name><surname>Miller</surname><given-names>Grant</given-names></name><xref ref-type="aff" rid="aff2">2</xref><xref ref-type="fn" rid="con12"/><xref ref-type="fn" rid="conf2"/></contrib><contrib contrib-type="author" id="author-207338"><name><surname>Lintott</surname><given-names>Chris</given-names></name><xref ref-type="aff" rid="aff2">2</xref><xref ref-type="fn" rid="con13"/><xref ref-type="fn" rid="conf2"/></contrib><contrib contrib-type="author" id="author-228808"><name><surname>Clifton</surname><given-names>David</given-names></name><xref ref-type="aff" rid="aff4">4</xref><xref ref-type="other" rid="fund1"/><xref ref-type="other" rid="fund2"/><xref ref-type="fn" rid="con14"/><xref ref-type="fn" rid="conf2"/></contrib><contrib contrib-type="author" id="author-94982"><name><surname>Crook</surname><given-names>Derrick W</given-names></name><contrib-id authenticated="true" contrib-id-type="orcid">https://orcid.org/0000-0002-0590-2850</contrib-id><xref ref-type="aff" rid="aff1">1</xref><xref ref-type="other" rid="fund1"/><xref ref-type="other" rid="fund2"/><xref ref-type="fn" rid="con15"/><xref ref-type="fn" rid="conf2"/></contrib><contrib contrib-type="author" id="author-94984"><name><surname>Walker</surname><given-names>A Sarah</given-names></name><contrib-id authenticated="true" contrib-id-type="orcid">https://orcid.org/0000-0002-0412-8509</contrib-id><xref ref-type="aff" rid="aff1">1</xref><xref ref-type="other" rid="fund1"/><xref ref-type="other" rid="fund2"/><xref ref-type="fn" rid="con16"/><xref ref-type="fn" rid="conf2"/></contrib><contrib contrib-type="author"><collab>The Zooniverse Volunteer Community</collab><xref ref-type="fn" rid="con17"/><xref ref-type="fn" rid="conf1"/></contrib><contrib contrib-type="author"><collab>The CRyPTIC Consortium<contrib-group><contrib><name><surname>Barilar</surname><given-names>Ivan</given-names></name><aff><institution>Research Center Borstel</institution><addr-line><named-content content-type="city">Borstel</named-content></addr-line><country>Germany</country></aff></contrib><contrib><name><surname>Battaglia</surname><given-names>Simone</given-names></name><aff><institution>IRCCS San Raffaele Scientific Institute</institution><addr-line><named-content content-type="city">Milan</named-content></addr-line><country>Italy</country></aff></contrib><contrib><name><surname>Borroni</surname><given-names>Emanuele</given-names></name><aff><institution>IRCCS San Raffaele Scientific Institute</institution><addr-line><named-content content-type="city">Milan</named-content></addr-line><country>Italy</country></aff></contrib><contrib><name><surname>Brandao</surname><given-names>Angela Pires</given-names></name><aff><institution>Oswaldo Cruz Foundation</institution><addr-line><named-content content-type="city">Rio de Janeiro</named-content></addr-line><country>Brazil</country></aff><aff><institution>Institute Adolfo Lutz</institution><addr-line><named-content content-type="city">Sao Paulo</named-content></addr-line><country>Brazil</country></aff></contrib><contrib><name><surname>Brankin</surname><given-names>Alice</given-names></name><aff><institution>University of Oxford</institution><addr-line><named-content content-type="city">Oxford</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Cabibbe</surname><given-names>Andrea Maurizio</given-names></name><aff><institution>IRCCS San Raffaele Scientific Institute</institution><addr-line><named-content content-type="city">Milan</named-content></addr-line><country>Italy</country></aff></contrib><contrib><name><surname>Carter</surname><given-names>Joshua</given-names></name><aff><institution>Stanford University School of Medicine</institution><addr-line><named-content content-type="city">Stanford</named-content></addr-line><country>United States</country></aff></contrib><contrib><name><surname>Cirillo</surname><given-names>Daniela Maria</given-names></name><aff><institution>IRCCS San Raffaele Scientific Institute</institution><addr-line><named-content content-type="city">Milan</named-content></addr-line><country>Italy</country></aff></contrib><contrib><name><surname>Claxton</surname><given-names>Pauline</given-names></name><aff><institution>Scottish Mycobacteria Reference Laboratory</institution><addr-line><named-content content-type="city">Edinburgh</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Clifton</surname><given-names>David A</given-names></name><aff><institution>University of Oxford</institution><addr-line><named-content content-type="city">Oxford</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Cohen</surname><given-names>Ted</given-names></name><aff><institution>Yale School of Public Health</institution><addr-line><named-content content-type="city">Yale</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Coronel</surname><given-names>Jorge</given-names></name><aff><institution>Universidad Peruana Cayetano Heredia</institution><addr-line><named-content content-type="city">Lima</named-content></addr-line><country>Peru</country></aff></contrib><contrib><name><surname>Crook</surname><given-names>Derrick W</given-names></name><aff><institution>University of Oxford</institution><addr-line><named-content content-type="city">Oxford</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Earle</surname><given-names>Sarah G</given-names></name><aff><institution>University of Oxford</institution><addr-line><named-content content-type="city">Oxford</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Escuyer</surname><given-names>Vincent</given-names></name><aff><institution>Wadsworth Center, New York State Department of Health</institution><addr-line><named-content content-type="city">Albany</named-content></addr-line><country>United States</country></aff></contrib><contrib><name><surname>Ferrazoli</surname><given-names>Lucilaine</given-names></name><aff><institution>Institute Adolfo Lutz</institution><addr-line><named-content content-type="city">Sao Paulo</named-content></addr-line><country>Brazil</country></aff></contrib><contrib><name><surname>Fowler</surname><given-names>Philip W</given-names></name><aff><institution>University of Oxford</institution><addr-line><named-content content-type="city">Oxford</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Gao</surname><given-names>George F</given-names></name><aff><institution>Chinese Center for Disease Control and Prevention</institution><addr-line><named-content content-type="city">Beijing</named-content></addr-line><country>China</country></aff></contrib><contrib><name><surname>Gardy</surname><given-names>Jennifer</given-names></name><aff><institution-wrap><institution-id institution-id-type="ror">https://ror.org/0456r8d26</institution-id><institution>Bill & Melinda Gates Foundation</institution></institution-wrap><addr-line><named-content content-type="city">Seattle</named-content></addr-line><country>United States</country></aff></contrib><contrib><name><surname>Gharbia</surname><given-names>Saheer</given-names></name><aff><institution>UK Health Security Agency</institution><addr-line><named-content content-type="city">London</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Ghisi</surname><given-names>Kelen Teixeira</given-names></name><aff><institution>Institute Adolfo Lutz</institution><addr-line><named-content content-type="city">Sao Paulo</named-content></addr-line><country>Brazil</country></aff></contrib><contrib><name><surname>Ghodousi</surname><given-names>Arash</given-names></name><aff><institution>IRCCS San Raffaele Scientific Institute</institution><addr-line><named-content content-type="city">Milan</named-content></addr-line><country>Italy</country></aff><aff><institution>Vita-Salute San Raffaele University</institution><addr-line><named-content content-type="city">Milan</named-content></addr-line><country>Italy</country></aff></contrib><contrib><name><surname>Gibertoni Cruz</surname><given-names>Ana Luıza</given-names></name><aff><institution>University of Oxford</institution><addr-line><named-content content-type="city">Oxford</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Grazian</surname><given-names>Clara</given-names></name><aff><institution>University of New South Wales</institution><addr-line><named-content content-type="city">Sydney</named-content></addr-line><country>Australia</country></aff></contrib><contrib><name><surname>Guthrie</surname><given-names>Jennifer L</given-names></name><aff><institution>The University of British Columbia</institution><addr-line><named-content content-type="city">Vancouver</named-content></addr-line><country>Canada</country></aff><aff><institution>Public Health Ontario</institution><addr-line><named-content content-type="city">Toronto</named-content></addr-line><country>Canada</country></aff></contrib><contrib><name><surname>He</surname><given-names>Wencong</given-names></name><aff><institution>Chinese Center for Disease Control and Prevention</institution><addr-line><named-content content-type="city">Beijing</named-content></addr-line><country>China</country></aff></contrib><contrib><name><surname>Hoffmann</surname><given-names>Harald</given-names></name><aff><institution>SYNLAB Gauting</institution><addr-line><named-content content-type="city">Munich</named-content></addr-line><country>Germany</country></aff><aff><institution>Institute of Microbiology and Laboratory Medicine,WHO-SRL Gauting</institution><addr-line><named-content content-type="city">IMLred</named-content></addr-line><country>Germany</country></aff></contrib><contrib><name><surname>Hoosdally</surname><given-names>Sarah J</given-names></name><aff><institution>University of Oxford</institution><addr-line><named-content content-type="city">Oxford</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Hunt</surname><given-names>Martin</given-names></name><aff><institution>University of Oxford</institution><addr-line><named-content content-type="city">Oxford</named-content></addr-line><country>United Kingdom</country></aff><aff><institution>EMBL-EBI</institution><addr-line><named-content content-type="city">Hinxton</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Iqbal</surname><given-names>Zamin</given-names></name><aff><institution>EMBL-EBI</institution><addr-line><named-content content-type="city">Hinxton</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Ismail</surname><given-names>Nazir Ahmed</given-names></name><aff><institution>National Institute for Communicable Diseases</institution><addr-line><named-content content-type="city">Johannesburg</named-content></addr-line><country>South Africa</country></aff></contrib><contrib><name><surname>Jarrett</surname><given-names>Lisa</given-names></name><aff><institution>UK Health Security Agency</institution><addr-line><named-content content-type="city">Birmingham</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Joseph</surname><given-names>Lavania</given-names></name><aff><institution>National Institute for Communicable Diseases</institution><addr-line><named-content content-type="city">Johannesburg</named-content></addr-line><country>South Africa</country></aff></contrib><contrib><name><surname>Jou</surname><given-names>Ruwen</given-names></name><aff><institution>Taiwan Centers for Disease Control</institution><addr-line><named-content content-type="city">Taipei</named-content></addr-line><country>Taiwan</country></aff></contrib><contrib><name><surname>Kambli</surname><given-names>Priti</given-names></name><aff><institution>Hinduja Hospital</institution><addr-line><named-content content-type="city">Mumbai</named-content></addr-line><country>India</country></aff></contrib><contrib><name><surname>Knaggs</surname><given-names>Jeff</given-names></name><aff><institution>University of Oxford</institution><addr-line><named-content content-type="city">Oxford</named-content></addr-line><country>United Kingdom</country></aff><aff><institution>EMBL-EBI</institution><addr-line><named-content content-type="city">Hinxton</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Koch</surname><given-names>Anastasia</given-names></name><aff><institution-wrap><institution-id institution-id-type="ror">https://ror.org/03p74gp79</institution-id><institution>University of Cape Town</institution></institution-wrap><addr-line><named-content content-type="city">Cape Town</named-content></addr-line><country>South Africa</country></aff></contrib><contrib><name><surname>Kohlerschmidt</surname><given-names>Donna</given-names></name><aff><institution>Wadsworth Center, New York State Department of Health</institution><addr-line><named-content content-type="city">Albany</named-content></addr-line><country>United States</country></aff></contrib><contrib><name><surname>Kouchaki</surname><given-names>Samaneh</given-names></name><aff><institution>University of Oxford</institution><addr-line><named-content content-type="city">Oxford</named-content></addr-line><country>United Kingdom</country></aff><aff><institution>University of Surrey</institution><addr-line><named-content content-type="city">Guildford</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Lachapelle</surname><given-names>Alexander S</given-names></name><aff><institution>University of Oxford</institution><addr-line><named-content content-type="city">Oxford</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Lalvani</surname><given-names>Ajit</given-names></name><aff><institution>Imperial College</institution><addr-line><named-content content-type="city">London</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Lapierre</surname><given-names>Simon Grandjean</given-names></name><aff><institution>Universite de Montreal</institution><addr-line><named-content content-type="city">Canada</named-content></addr-line><country>United States</country></aff></contrib><contrib><name><surname>Laurenson</surname><given-names>Ian F</given-names></name><aff><institution>Scottish Mycobacteria Reference Laboratory</institution><addr-line><named-content content-type="city">Edinburgh</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Letcher</surname><given-names>Brice</given-names></name><aff><institution>EMBL-EBI</institution><addr-line><named-content content-type="city">Hinxton</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Lin</surname><given-names>Wan-Hsuan</given-names></name><aff><institution>Taiwan Centers for Disease Control</institution><addr-line><named-content content-type="city">Taipei</named-content></addr-line><country>Taiwan</country></aff></contrib><contrib><name><surname>Liu</surname><given-names>Chunfa</given-names></name><aff><institution>Chinese Center for Disease Control and Prevention</institution><addr-line><named-content content-type="city">Beijing</named-content></addr-line><country>China</country></aff></contrib><contrib><name><surname>Liu</surname><given-names>Dongxin</given-names></name><aff><institution>Chinese Center for Disease Control and Prevention</institution><addr-line><named-content content-type="city">Beijing</named-content></addr-line><country>China</country></aff></contrib><contrib><name><surname>Malone</surname><given-names>Kerri M</given-names></name><aff><institution>EMBL-EBI</institution><addr-line><named-content content-type="city">Hinxton</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Mandal</surname><given-names>Ayan</given-names></name><aff><institution>The Foundation for Medical Research</institution><addr-line><named-content content-type="city">Mumbai</named-content></addr-line><country>India</country></aff></contrib><contrib><name><surname>Matias</surname><given-names>Daniela</given-names></name><aff><institution>UK Health Security Agency</institution><addr-line><named-content content-type="city">Birmingham</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Meintjes</surname><given-names>Graeme</given-names></name><aff><institution-wrap><institution-id institution-id-type="ror">https://ror.org/03p74gp79</institution-id><institution>University of Cape Town</institution></institution-wrap><addr-line><named-content content-type="city">Cape Town</named-content></addr-line><country>South Africa</country></aff></contrib><contrib><name><surname>Freitas Mendes</surname><given-names>Flavia</given-names></name><aff><institution>Institute Adolfo Lutz</institution><addr-line><named-content content-type="city">Sao Paulo</named-content></addr-line><country>Brazil</country></aff></contrib><contrib><name><surname>Merker</surname><given-names>Matthias</given-names></name><aff><institution>Research Center Borstel</institution><addr-line><named-content content-type="city">Borstel</named-content></addr-line><country>Germany</country></aff></contrib><contrib><name><surname>Mihalic</surname><given-names>Marina</given-names></name><aff><institution>Institute of Microbiology and Laboratory Medicine,WHO-SRL Gauting</institution><addr-line><named-content content-type="city">IMLred</named-content></addr-line><country>Germany</country></aff></contrib><contrib><name><surname>Millard</surname><given-names>James</given-names></name><aff><institution>Africa Health Research Institute</institution><addr-line><named-content content-type="city">Durban</named-content></addr-line><country>South Africa</country></aff></contrib><contrib><name><surname>Miotto</surname><given-names>Paolo</given-names></name><aff><institution>IRCCS San Raffaele Scientific Institute</institution><addr-line><named-content content-type="city">Milan</named-content></addr-line><country>Italy</country></aff></contrib><contrib><name><surname>Mistry</surname><given-names>Nerges</given-names></name><aff><institution>The Foundation for Medical Research</institution><addr-line><named-content content-type="city">Mumbai</named-content></addr-line><country>India</country></aff></contrib><contrib><name><surname>Moore</surname><given-names>David</given-names></name><aff><institution>Universidad Peruana Cayetano Heredia</institution><addr-line><named-content content-type="city">Lima</named-content></addr-line><country>Peru</country></aff><aff><institution>London School of Hygiene and Tropical Medicine</institution><addr-line><named-content content-type="city">London</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Dreyer</surname><given-names>Viola</given-names></name><aff><institution>Research Center Borstel</institution><addr-line><named-content content-type="city">Borstel</named-content></addr-line><country>Germany</country></aff></contrib><contrib><name><surname>Chetty</surname><given-names>Darren</given-names></name><aff><institution>Africa Health Research Institute</institution><addr-line><named-content content-type="city">Durban</named-content></addr-line><country>South Africa</country></aff></contrib><contrib><name><surname>Musser</surname><given-names>Kimberlee A</given-names></name><aff><institution>Wadsworth Center, New York State Department of Health</institution><addr-line><named-content content-type="city">Albany</named-content></addr-line><country>United States</country></aff></contrib><contrib><name><surname>Ngcamu</surname><given-names>Dumisani</given-names></name><aff><institution>National Institute for Communicable Diseases</institution><addr-line><named-content content-type="city">Johannesburg</named-content></addr-line><country>South Africa</country></aff></contrib><contrib><name><surname>Nhung</surname><given-names>Hoang Ngoc</given-names></name><aff><institution>Oxford University Clinical Research Unit</institution><addr-line><named-content content-type="city">Ho Chi Minh City</named-content></addr-line><country>Viet Nam</country></aff></contrib><contrib><name><surname>Grandjean</surname><given-names>Louis</given-names></name><aff><institution>University College London</institution><addr-line><named-content content-type="city">London</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Nilgiriwala</surname><given-names>Kayzad Soli</given-names></name><aff><institution>The Foundation for Medical Research</institution><addr-line><named-content content-type="city">Mumbai</named-content></addr-line><country>India</country></aff></contrib><contrib><name><surname>Nimmo</surname><given-names>Camus</given-names></name><aff><institution>University College London</institution><addr-line><named-content content-type="city">London</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Okozi</surname><given-names>Nana</given-names></name><aff><institution>National Institute for Communicable Diseases</institution><addr-line><named-content content-type="city">Johannesburg</named-content></addr-line><country>South Africa</country></aff></contrib><contrib><name><surname>Oliveira</surname><given-names>Rosangela Siqueira</given-names></name><aff><institution>Institute Adolfo Lutz</institution><addr-line><named-content content-type="city">Sao Paulo</named-content></addr-line><country>Brazil</country></aff></contrib><contrib><name><surname>Omar</surname><given-names>Shaheed Vally</given-names></name><aff><institution>National Institute for Communicable Diseases</institution><addr-line><named-content content-type="city">Johannesburg</named-content></addr-line><country>South Africa</country></aff></contrib><contrib><name><surname>Paton</surname><given-names>Nicholas</given-names></name><aff><institution-wrap><institution-id institution-id-type="ror">https://ror.org/01tgyzw49</institution-id><institution>National University of Singapore</institution></institution-wrap><addr-line><named-content content-type="city">Singapore</named-content></addr-line><country>Singapore</country></aff></contrib><contrib><name><surname>Peto</surname><given-names>Timothy EA</given-names></name><aff><institution>University of Oxford</institution><addr-line><named-content content-type="city">Oxford</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Watanabe Pinhata</surname><given-names>Juliana Maira</given-names></name><aff><institution>Institute Adolfo Lutz</institution><addr-line><named-content content-type="city">Sao Paulo</named-content></addr-line><country>Brazil</country></aff></contrib><contrib><name><surname>Plesnik</surname><given-names>Sara</given-names></name><aff><institution>Institute of Microbiology and Laboratory Medicine,WHO-SRL Gauting</institution><addr-line><named-content content-type="city">IMLred</named-content></addr-line><country>Germany</country></aff></contrib><contrib><name><surname>Puyen</surname><given-names>Zully M</given-names></name><aff><institution>Instituto Nacional de Salud</institution><addr-line><named-content content-type="city">Lima</named-content></addr-line><country>Peru</country></aff></contrib><contrib><name><surname>Rabodoarivelo</surname><given-names>Marie Sylvianne</given-names></name><aff><institution>Institut Pasteur de Madagascar</institution><addr-line><named-content content-type="city">Antananarivo</named-content></addr-line><country>Madagascar</country></aff></contrib><contrib><name><surname>Rakotosamimanana</surname><given-names>Niaina</given-names></name><aff><institution>Institut Pasteur de Madagascar</institution><addr-line><named-content content-type="city">Antananarivo</named-content></addr-line><country>Madagascar</country></aff></contrib><contrib><name><surname>Rancoita</surname><given-names>Paola MV</given-names></name><aff><institution>Vita-Salute San Raffaele University</institution><addr-line><named-content content-type="city">Milan</named-content></addr-line><country>Italy</country></aff></contrib><contrib><name><surname>Rathod</surname><given-names>Priti</given-names></name><aff><institution>UK Health Security Agency</institution><addr-line><named-content content-type="city">Birmingham</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Robinson</surname><given-names>Esther</given-names></name><aff><institution>UK Health Security Agency</institution><addr-line><named-content content-type="city">Birmingham</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Rodger</surname><given-names>Gillian</given-names></name><aff><institution>University of Oxford</institution><addr-line><named-content content-type="city">Oxford</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Rodrigues</surname><given-names>Camilla</given-names></name><aff><institution>Hinduja Hospital</institution><addr-line><named-content content-type="city">Mumbai</named-content></addr-line><country>India</country></aff></contrib><contrib><name><surname>Rodwell</surname><given-names>Timothy C</given-names></name><aff><institution>FIND</institution><addr-line><named-content content-type="city">Geneva</named-content></addr-line><country>Switzerland</country></aff><aff><institution>University of California, San Diego</institution><addr-line><named-content content-type="city">San Diego</named-content></addr-line><country>United States</country></aff></contrib><contrib><name><surname>Roohi</surname><given-names>Aysha</given-names></name><aff><institution>University of Oxford</institution><addr-line><named-content content-type="city">Oxford</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Santos-Lazaro</surname><given-names>David</given-names></name><aff><institution>Instituto Nacional de Salud</institution><addr-line><named-content content-type="city">Lima</named-content></addr-line><country>Peru</country></aff></contrib><contrib><name><surname>Shah</surname><given-names>Sanchi</given-names></name><aff><institution>The Foundation for Medical Research</institution><addr-line><named-content content-type="city">Mumbai</named-content></addr-line><country>India</country></aff></contrib><contrib><name><surname>Kohl</surname><given-names>Thomas Andreas</given-names></name><aff><institution>Research Center Borstel</institution><addr-line><named-content content-type="city">Borstel</named-content></addr-line><country>Germany</country></aff></contrib><contrib><name><surname>Smith</surname><given-names>Grace</given-names></name><aff><institution>UK Health Security Agency</institution><addr-line><named-content content-type="city">London</named-content></addr-line><country>United Kingdom</country></aff><aff><institution>UK Health Security Agency</institution><addr-line><named-content content-type="city">Birmingham</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Solano</surname><given-names>Walter</given-names></name><aff><institution>Universidad Peruana Cayetano Heredia</institution><addr-line><named-content content-type="city">Lima</named-content></addr-line><country>Peru</country></aff></contrib><contrib><name><surname>Spitaleri</surname><given-names>Andrea</given-names></name><aff><institution>IRCCS San Raffaele Scientific Institute</institution><addr-line><named-content content-type="city">Milan</named-content></addr-line><country>Italy</country></aff><aff><institution>Vita-Salute San Raffaele University</institution><addr-line><named-content content-type="city">Milan</named-content></addr-line><country>Italy</country></aff></contrib><contrib><name><surname>Supply</surname><given-names>Philip</given-names></name><aff><institution>Institut Pasteur de Lille</institution><addr-line><named-content content-type="city">Lille</named-content></addr-line><country>France</country></aff></contrib><contrib><name><surname>Steyn</surname><given-names>Adrie JC</given-names></name><aff><institution>Africa Health Research Institute</institution><addr-line><named-content content-type="city">Durban</named-content></addr-line><country>South Africa</country></aff></contrib><contrib><name><surname>Surve</surname><given-names>Utkarsha</given-names></name><aff><institution>Hinduja Hospital</institution><addr-line><named-content content-type="city">Mumbai</named-content></addr-line><country>India</country></aff></contrib><contrib><name><surname>Tahseen</surname><given-names>Sabira</given-names></name><aff><institution>National TB Reference Laboratory, National TB Control Program</institution><addr-line><named-content content-type="city">Islamabad</named-content></addr-line><country>Pakistan</country></aff></contrib><contrib><name><surname>Thuong</surname><given-names>Nguyen Thuy</given-names></name><aff><institution>Oxford University Clinical Research Unit</institution><addr-line><named-content content-type="city">Ho Chi Minh City</named-content></addr-line><country>Viet Nam</country></aff></contrib><contrib><name><surname>Thwaites</surname><given-names>Guy</given-names></name><aff><institution>University of Oxford</institution><addr-line><named-content content-type="city">Oxford</named-content></addr-line><country>United Kingdom</country></aff><aff><institution>Oxford University Clinical Research Unit</institution><addr-line><named-content content-type="city">Ho Chi Minh City</named-content></addr-line><country>Viet Nam</country></aff></contrib><contrib><name><surname>Todt</surname><given-names>Katharina</given-names></name><aff><institution>Institute of Microbiology and Laboratory Medicine,WHO-SRL Gauting</institution><addr-line><named-content content-type="city">IMLred</named-content></addr-line><country>Germany</country></aff></contrib><contrib><name><surname>Trovato</surname><given-names>Alberto</given-names></name><aff><institution>IRCCS San Raffaele Scientific Institute</institution><addr-line><named-content content-type="city">Milan</named-content></addr-line><country>Italy</country></aff></contrib><contrib><name><surname>Utpatel</surname><given-names>Christian</given-names></name><aff><institution>Research Center Borstel</institution><addr-line><named-content content-type="city">Borstel</named-content></addr-line><country>Germany</country></aff></contrib><contrib><name><surname>Van Rie</surname><given-names>Annelies</given-names></name><aff><institution-wrap><institution-id institution-id-type="ror">https://ror.org/008x57b05</institution-id><institution>University of Antwerp</institution></institution-wrap><addr-line><named-content content-type="city">Antwerp</named-content></addr-line><country>Belgium</country></aff></contrib><contrib><name><surname>Vijay</surname><given-names>Srinivasan</given-names></name><aff><institution-wrap><institution-id institution-id-type="ror">https://ror.org/01nrxwf90</institution-id><institution>University of Edinburgh</institution></institution-wrap><addr-line><named-content content-type="city">Edinburgh</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Walker</surname><given-names>Timothy M</given-names></name><aff><institution>University of Oxford</institution><addr-line><named-content content-type="city">Oxford</named-content></addr-line><country>United Kingdom</country></aff><aff><institution>University of Oxford</institution><addr-line><named-content content-type="city">Oxford</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Walker</surname><given-names>A Sarah</given-names></name><aff><institution>University of Oxford</institution><addr-line><named-content content-type="city">Oxford</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Warren</surname><given-names>Robin</given-names></name><aff><institution>Stellenbosch University</institution><addr-line><named-content content-type="city">Cape Town</named-content></addr-line><country>South Africa</country></aff></contrib><contrib><name><surname>Werngren</surname><given-names>Jim</given-names></name><aff><institution-wrap><institution-id institution-id-type="ror">https://ror.org/05x4m5564</institution-id><institution>Public Health Agency of Sweden</institution></institution-wrap><addr-line><named-content content-type="city">Solna</named-content></addr-line><country>Sweden</country></aff></contrib><contrib><name><surname>Groenheit</surname><given-names>Ramona</given-names></name><aff><institution-wrap><institution-id institution-id-type="ror">https://ror.org/05x4m5564</institution-id><institution>Public Health Agency of Sweden</institution></institution-wrap><addr-line><named-content content-type="city">Solna</named-content></addr-line><country>Sweden</country></aff></contrib><contrib><name><surname>Wijkander</surname><given-names>Maria</given-names></name><aff><institution-wrap><institution-id institution-id-type="ror">https://ror.org/05x4m5564</institution-id><institution>Public Health Agency of Sweden</institution></institution-wrap><addr-line><named-content content-type="city">Solna</named-content></addr-line><country>Sweden</country></aff></contrib><contrib><name><surname>Wilkinson</surname><given-names>Robert J</given-names></name><aff><institution>Imperial College</institution><addr-line><named-content content-type="city">London</named-content></addr-line><country>United Kingdom</country></aff><aff><institution>Wellcome Centre for Infectious Diseases Research in Africa</institution><addr-line><named-content content-type="city">Cape Town</named-content></addr-line><country>South Africa</country></aff><aff><institution>Francis Crick Institute</institution><addr-line><named-content content-type="city">London</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Wilson</surname><given-names>Daniel J</given-names></name><aff><institution>University of Oxford</institution><addr-line><named-content content-type="city">Oxford</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Wintringer</surname><given-names>Penelope</given-names></name><aff><institution>EMBL-EBI</institution><addr-line><named-content content-type="city">Hinxton</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Xiao</surname><given-names>Yu-Xin</given-names></name><aff><institution>Taiwan Centers for Disease Control</institution><addr-line><named-content content-type="city">Taipei</named-content></addr-line><country>Taiwan</country></aff></contrib><contrib><name><surname>Yang</surname><given-names>Yang</given-names></name><aff><institution>University of Oxford</institution><addr-line><named-content content-type="city">Oxford</named-content></addr-line><country>United Kingdom</country></aff></contrib><contrib><name><surname>Yanlin</surname><given-names>Zhao</given-names></name><aff><institution>Chinese Center for Disease Control and Prevention</institution><addr-line><named-content content-type="city">Beijing</named-content></addr-line><country>China</country></aff></contrib><contrib><name><surname>Yao</surname><given-names>Shen-Yuan</given-names></name><aff><institution>National Institute for Communicable Diseases</institution><addr-line><named-content content-type="city">Johannesburg</named-content></addr-line><country>South Africa</country></aff></contrib><contrib><name><surname>Zhu</surname><given-names>Baoli</given-names></name><aff><institution-wrap><institution-id institution-id-type="ror">https://ror.org/034t30j35</institution-id><institution>Institute of Microbiology, Chinese Academy of Sciences</institution></institution-wrap><addr-line><named-content content-type="city">Beijing</named-content></addr-line><country>China</country></aff></contrib><contrib><name><surname>Niemann</surname><given-names>Stefan</given-names></name><aff><institution>Research Center Borstel</institution><addr-line><named-content content-type="city">Borstel</named-content></addr-line><country>Germany</country></aff><aff><institution>German Center for Infection Research (DZIF)</institution><addr-line><named-content content-type="city">Hamburg-Lubeck-Borstel-Riems</named-content></addr-line><country>Germany</country></aff></contrib><contrib><name><surname>O'Donnell</surname><given-names>Max</given-names></name><aff><institution-wrap><institution-id institution-id-type="ror">https://ror.org/01esghr10</institution-id><institution>Columbia University Irving Medical Center</institution></institution-wrap><addr-line><named-content content-type="city">New York</named-content></addr-line><country>United States</country></aff></contrib></contrib-group></collab><xref ref-type="fn" rid="con18"/><xref ref-type="fn" rid="conf1"/></contrib><aff id="aff1"><label>1</label><institution-wrap><institution-id institution-id-type="ror">https://ror.org/052gg0110</institution-id><institution>Nuffield Department of Medicine, University of Oxford</institution></institution-wrap><addr-line><named-content content-type="city">Oxford</named-content></addr-line><country>United Kingdom</country></aff><aff id="aff2"><label>2</label><institution-wrap><institution-id institution-id-type="ror">https://ror.org/052gg0110</institution-id><institution>Zooniverse, Department of Physics, University of Oxford</institution></institution-wrap><addr-line><named-content content-type="city">Oxford</named-content></addr-line><country>United Kingdom</country></aff><aff id="aff3"><label>3</label><institution-wrap><institution-id institution-id-type="ror">https://ror.org/04tnbqb63</institution-id><institution>Electron Microscopy Science Technology Platform, The Francis Crick Institute</institution></institution-wrap><addr-line><named-content content-type="city">London</named-content></addr-line><country>United Kingdom</country></aff><aff id="aff4"><label>4</label><institution-wrap><institution-id institution-id-type="ror">https://ror.org/052gg0110</institution-id><institution>Institute of Biomedical Engineering, University of Oxford</institution></institution-wrap><addr-line><named-content content-type="city">Oxford</named-content></addr-line><country>United Kingdom</country></aff><aff id="aff5"><label>5</label><institution-wrap><institution-id institution-id-type="ror">https://ror.org/052gg0110</institution-id><institution>Citizen Scientist, c/o Zooniverse, Department of Physics, University of Oxford</institution></institution-wrap><addr-line><named-content content-type="city">Oxford</named-content></addr-line><country>United Kingdom</country></aff></contrib-group><contrib-group content-type="section"><contrib contrib-type="editor"><name><surname>Kana</surname><given-names>Bavesh D</given-names></name><role>Reviewing Editor</role><aff><institution-wrap><institution-id institution-id-type="ror">https://ror.org/03rp50x72</institution-id><institution>University of the Witwatersrand</institution></institution-wrap><country>South Africa</country></aff></contrib><contrib contrib-type="senior_editor"><name><surname>Kana</surname><given-names>Bavesh D</given-names></name><role>Senior Editor</role><aff><institution-wrap><institution-id institution-id-type="ror">https://ror.org/03rp50x72</institution-id><institution>University of the Witwatersrand</institution></institution-wrap><country>South Africa</country></aff></contrib></contrib-group><pub-date publication-format="electronic" date-type="publication"><day>19</day><month>05</month><year>2022</year></pub-date><pub-date pub-type="collection"><year>2022</year></pub-date><volume>11</volume><elocation-id>e75046</elocation-id><history><date date-type="received" iso-8601-date="2021-10-28"><day>28</day><month>10</month><year>2021</year></date><date date-type="accepted" iso-8601-date="2022-05-15"><day>15</day><month>05</month><year>2022</year></date></history><pub-history><event><event-desc>This manuscript was published as a preprint at bioRxiv.</event-desc><date date-type="preprint" iso-8601-date="2021-07-21"><day>21</day><month>07</month><year>2021</year></date><self-uri content-type="preprint" xlink:href="https://doi.org/10.1101/2021.07.20.453060"/></event></pub-history><permissions><copyright-statement>© 2022, Fowler et al</copyright-statement><copyright-year>2022</copyright-year><copyright-holder>Fowler et al</copyright-holder><ali:free_to_read/><license xlink:href="http://creativecommons.org/licenses/by/4.0/"><ali:license_ref>http://creativecommons.org/licenses/by/4.0/</ali:license_ref><license-p>This article is distributed under the terms of the <ext-link ext-link-type="uri" xlink:href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution License</ext-link>, which permits unrestricted use and redistribution provided that the original author and source are credited.</license-p></license></permissions><self-uri content-type="pdf" xlink:href="elife-75046-v3.pdf"/><self-uri content-type="figures-pdf" xlink:href="elife-75046-figures-v3.pdf"/><abstract><p>Tuberculosis is a respiratory disease that is treatable with antibiotics. An increasing prevalence of resistance means that to ensure a good treatment outcome it is desirable to test the susceptibility of each infection to different antibiotics. Conventionally, this is done by culturing a clinical sample and then exposing aliquots to a panel of antibiotics, each being present at a pre-determined concentration, thereby determining if the sample isresistant or susceptible to each sample. The minimum inhibitory concentration (MIC) of a drug is the lowestconcentration that inhibits growth and is a more useful quantity but requires each sample to be tested at a range ofconcentrations for each drug. Using 96-well broth micro dilution plates with each well containing a lyophilised pre-determined amount of an antibiotic is a convenient and cost-effective way to measure the MICs of several drugs at once for a clinical sample. Although accurate, this is still an expensive and slow process that requires highly-skilled and experienced laboratory scientists. Here we show that, through the BashTheBug project hosted on the Zooniverse citizen science platform, a crowd of volunteers can reproducibly and accurately determine the MICs for 13 drugs and that simply taking the median or mode of 11–17 independent classifications is sufficient. There is therefore a potential role for crowds to support (but not supplant) the role of experts in antibiotic susceptibility testing.</p></abstract><abstract abstract-type="executive-summary"><title>eLife digest</title><p>Tuberculosis is a bacterial respiratory infection that kills about 1.4 million people worldwide each year. While antibiotics can cure the condition, the bacterium responsible for this disease, <italic>Mycobacterium tuberculosis,</italic> is developing resistance to these treatments. Choosing which antibiotics to use to treat the infection more carefully may help to combat the growing threat of drug-resistant bacteria.</p><p>One way to find the best choice is to test how an antibiotic affects the growth of <italic>M. tuberculosis</italic> in the laboratory. To speed up this process, laboratories test multiple drugs simultaneously. They do this by growing bacteria on plates with 96 wells and injecting individual antibiotics in to each well at different concentrations.</p><p>The Comprehensive Resistance Prediction for Tuberculosis (CRyPTIC) consortium has used this approach to collect and analyse bacteria from over 20,000 tuberculosis patients. An image of the 96-well plate is then captured and the level of bacterial growth in each well is assessed by laboratory scientists. But this work is difficult, time-consuming, and subjective, even for tuberculosis experts. Here, Fowler et al. show that enlisting citizen scientists may help speed up this process and reduce errors that arise from analysing such a large dataset.</p><p>In April 2017, Fowler et al. launched the project ‘BashTheBug’ on the Zooniverse citizen science platform where anyone can access and analyse the images from the CRyPTIC consortium. They found that a crowd of inexperienced volunteers were able to consistently and accurately measure the concentration of antibiotics necessary to inhibit the growth of <italic>M. tuberculosis.</italic> If the concentration is above a pre-defined threshold, the bacteria are considered to be resistant to the treatment. A consensus result could be reached by calculating the median value of the classifications provided by as few as 17 different BashTheBug participants.</p><p>The work of BashTheBug volunteers has reduced errors in the CRyPTIC project data, which has been used for several other studies. For instance, the World Health Organization (WHO) has also used the data to create a catalogue of genetic mutations associated with antibiotics resistance in <italic>M. tuberculosis</italic>. Enlisting citizen scientists has accelerated research on tuberculosis and may help with other pressing public health concerns.</p></abstract><kwd-group kwd-group-type="author-keywords"><kwd>M. tuberculosis</kwd><kwd>citizen science</kwd><kwd>clinical microbiology</kwd><kwd>tuberculosis</kwd><kwd>antibiotics</kwd></kwd-group><kwd-group kwd-group-type="research-organism"><title>Research organism</title><kwd>Other</kwd></kwd-group><funding-group><award-group id="fund1"><funding-source><institution-wrap><institution-id institution-id-type="FundRef">http://dx.doi.org/10.13039/100004440</institution-id><institution>Wellcome Trust</institution></institution-wrap></funding-source><award-id>200205/Z/15/Z</award-id><principal-award-recipient><name><surname>Fowler</surname><given-names>Philip W</given-names></name><name><surname>Wright</surname><given-names>Carla</given-names></name><name><surname>Hoosdally</surname><given-names>Sarah W</given-names></name><name><surname>Gibertoni Cruz</surname><given-names>Ana L</given-names></name><name><surname>Roohi</surname><given-names>Aysha</given-names></name><name><surname>Kouchaki</surname><given-names>Samaneh</given-names></name><name><surname>Walker</surname><given-names>Timothy M</given-names></name><name><surname>Peto</surname><given-names>Timothy EA</given-names></name><name><surname>Clifton</surname><given-names>David</given-names></name><name><surname>Crook</surname><given-names>Derrick W</given-names></name><name><surname>Walker</surname><given-names>A Sarah</given-names></name></principal-award-recipient></award-group><award-group id="fund2"><funding-source><institution-wrap><institution-id institution-id-type="FundRef">http://dx.doi.org/10.13039/100000865</institution-id><institution>Bill and Melinda Gates Foundation</institution></institution-wrap></funding-source><award-id>OPP1133541</award-id><principal-award-recipient><name><surname>Fowler</surname><given-names>Philip W</given-names></name><name><surname>Wright</surname><given-names>Carla</given-names></name><name><surname>Hoosdally</surname><given-names>Sarah W</given-names></name><name><surname>Gibertoni Cruz</surname><given-names>Ana L</given-names></name><name><surname>Roohi</surname><given-names>Aysha</given-names></name><name><surname>Kouchaki</surname><given-names>Samaneh</given-names></name><name><surname>Walker</surname><given-names>Timothy M</given-names></name><name><surname>Peto</surname><given-names>Timothy EA</given-names></name><name><surname>Clifton</surname><given-names>David</given-names></name><name><surname>Crook</surname><given-names>Derrick W</given-names></name><name><surname>Walker</surname><given-names>A Sarah</given-names></name></principal-award-recipient></award-group><award-group id="fund3"><funding-source><institution-wrap><institution-id institution-id-type="FundRef">http://dx.doi.org/10.13039/100004440</institution-id><institution>Wellcome Trust</institution></institution-wrap></funding-source><award-id>203141/Z/16/Z</award-id><principal-award-recipient><name><surname>Fowler</surname><given-names>Philip W</given-names></name></principal-award-recipient></award-group><funding-statement>The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication. For the purpose of Open Access, the authors have applied a CC BY public copyright license to any Author Accepted Manuscript version arising from this submission.</funding-statement></funding-group><custom-meta-group><custom-meta specific-use="meta-only"><meta-name>Author impact statement</meta-name><meta-value>A crowd of inexperienced volunteers can reproducibily and accurately measured how effective a panel of antibiotics are in treating a <italic>M. tuberculosis</italic> sample.</meta-value></custom-meta></custom-meta-group></article-meta></front><body><sec id="s1" sec-type="intro"><title>Introduction</title><p>Tuberculosis (TB) is a treatable (primarily) respiratory disease that caused illness in ten million people in 2019, with 1.4million deaths (<xref ref-type="bibr" rid="bib27">World Health Organization, 2020</xref>). Ordinarily this is more than any other single pathogen, although SARS-CoV-2 killed more people than TB in 2020 and is likely to do so again in 2021. Like most bacterial diseases treated with antibiotics, an increasing proportion of TB cases are resistant to one or more drugs.</p><p>Tackling this ‘silent pandemic’ will require action on several fronts, including the development of new antibiotics, better stewardship of existing antibiotics and much wider use of antibiotic susceptibility testing (AST) to guide prescribing decisions (<xref ref-type="bibr" rid="bib15">O’Neill, 2016</xref>). The prevailing AST paradigm is <italic>culture-based</italic>: a sample taken from the patient is grown and the pathogen identified. If required, further samples are cultured in the presence of different antibiotics and each test is inspected/measured to see which compounds inhibit the growth of the bacterium. A scientific laboratory, with an enhanced biosafety level and staffed by experienced and highly trained laboratory scientists, is required to carry out such AST. Compared to most pathogenic bacteria, <italic>M. tuberculosis</italic> is unusually difficult,mainly because its doubling time is so slow resulting in culture times being weeks rather than days. Maintaining such laboratories with a cadre of expert scientists is expensive and hence they tend to be found only at larger hospitals and national public health agencies, even in high-income countries. This model, whilst effective, is practically and economically difficult to scale up which explains in part why most antibiotic prescribing decisions are still done without any AST data.</p><p>The <italic>minimum inhibitory concentration</italic> (MIC) of an antibiotic is the lowest concentration that prevents growth of the pathogen. Mycobacterial samples are usually inoculated onto an appropriate growth medium that contains an antibiotic at a single critical concentration; if the MIC of the same is above the critical concentration it will grow and hence be labelled resistant, otherwise there is no growth and it will be classified as susceptible. Although simple, the accuracy of such an approach decreases when the expected MIC is of the same order as the critical concentration, as is the case for many antituberculars (<xref ref-type="bibr" rid="bib2">CRyPTIC Consortium, 2022</xref>). Historically, it has been assumed that, since accuracy is paramount, only highly-trained and experienced laboratory scientists (experts), or more recently, extensively-validated automatic algorithms part of accredited AST devices, can assess if a pathogen is growing and hence determine whether the sample is resistant or not.</p><p>In this paper, we shall show that a crowd of volunteers, who have no microbiological training, can reproducibly and accurately determine the growth of <italic>M. tuberculosis</italic> (the causative agent of TB) on a 96-well plate and thence the MICs for 13 different antibiotics. This is an application of the “wisdom of crowds” which can be traced back to observations made by <xref ref-type="bibr" rid="bib8">Galton, 1907</xref>. An even earlier use is the Germanic definition of the foot as one sixteenth the combined length of the left feet of the first sixteen men to leave a church on a Sunday (<xref ref-type="bibr" rid="bib10">Koebel, 1563</xref>). The main goal of the BashTheBug citizen science project, which was launched in April 2017 on the Zooniverse platform, is to reduce MIC measurement error in the large dataset of 20,637 clinical <italic>M. tuberculosis</italic> isolates collected by the Comprehensive Resistance Prediction for Tuberculosis: an International Consortium (CRyPTIC) project.</p></sec><sec id="s2" sec-type="results"><title>Results</title><sec id="s2-1"><title>Project launch and progress</title><p>After a successful beta-test on 22 March 2017, BashTheBug was launched as a project on the Zooniverse citizen science platform on 8 April 2017. The launch was publicised on social media, mainly Twitter, and mentioned on several websites and the Zooniverse users were notified via email. By the end of the first week, 2029 people had participated (of which 1,259 had usernames) classifying a total of 74,949 images – this includes the beta-test. The initial set of images were completed on 11 October 2017 and a second set was classified between 8 June 2020 and 15 November 2020 (<xref ref-type="fig" rid="fig1">Figure 1A</xref>) to increase the number of classifications per drug image from 11 to 17.</p><fig-group><fig id="fig1" position="float"><label>Figure 1.</label><caption><title>This dataset of 778,202 classifications was collected in two batches between April 2017 and Sep 2020 by 9372 volunteers.</title><p>(<bold>A</bold>) The classifications were done by the volunteers in two distinct batches; one during 2017 and a later one in 2020. Note that the higher participation during 2020 was due to the national restrictions imposed due to the SARS-Cov-2 pandemic. (<bold>B</bold>) The number of active users per day varied from zero to over 150. (C) The Lorenz curve demonstrates that there is considerable participation inequality in the project resulting in a Gini-coefficient of 0.85. (D) Volunteers spent different lengths of time classifying drug images after 14 days of incubation with a mode duration of 3.5 s.</p></caption><graphic mimetype="image" mime-subtype="jpeg" xlink:href="elife-75046.xml.media/fig1.jpg"/></fig><fig id="fig1s1" position="float" specific-use="child-fig"><label>Figure 1—figure supplement 1.</label><caption><title>Thank you to all the volunteers who contributed one or more classifications to this manuscript.</title><p>There are the 5810 usernames of all the volunteers in this montage – volunteers who did not register or sign in are not included.</p></caption><graphic mimetype="image" mime-subtype="jpeg" xlink:href="elife-75046.xml.media/fig1-figsupp1.jpg"/></fig><fig id="fig1s2" position="float" specific-use="child-fig"><label>Figure 1—figure supplement 2.</label><caption><title>The time spent by volunteers on each classification varied with a mode of 3.5 s.</title><p>Since one would expect different amounts of bacterial growth on the microdilution plates after (<bold>A</bold>) 7, (<bold>B</bold>) 10, (<bold>C</bold>) 14 and (<bold>D</bold>) 21 days the distributions of these were examined separately. All were, however, similar indicating that this did not have a significant effect.</p></caption><graphic mimetype="image" mime-subtype="jpeg" xlink:href="elife-75046.xml.media/fig1-figsupp2.jpg"/></fig><fig id="fig1s3" position="float" specific-use="child-fig"><label>Figure 1—figure supplement 3.</label><caption><title>The time spent by volunteers on each classification varied depending on the drug being considered.</title><p>The mode of each distribution is labelled. The drug the volunteers spent the longest on (bedaquiline, mode 4.8 s) was also one of those with the largest number (8) of wells. As measured by its mode of 3.2 s, the volunteers spent the least time classifying delamanid.</p></caption><graphic mimetype="image" mime-subtype="jpeg" xlink:href="elife-75046.xml.media/fig1-figsupp3.jpg"/></fig><fig id="fig1s4" position="float" specific-use="child-fig"><label>Figure 1—figure supplement 4.</label><caption><title>Every new user is shown this tutorial when they first join the BashTheBug Zooniverse project.</title><p>It uses example images to explain the task and then each of the options that they can choose to classify a drug image.</p></caption><graphic mimetype="image" mime-subtype="jpeg" xlink:href="elife-75046.xml.media/fig1-figsupp4.jpg"/></fig></fig-group></sec><sec id="s2-2"><title>Volunteers</title><p>In total, 9372 volunteers participated in classifying this dataset doing a total of 778,202 classifications (<xref ref-type="fig" rid="fig1s1">Figure 1—figure supplement 1</xref>). The number of citizen scientists is an over-estimate since users who did not register with the Zooniverse (and therefore could not be identified through their unique username) but did more than one session will be counted multiple times. On 251 of the 356 days (70 %) when these data were being collected there were ten or more volunteers active on the project (<xref ref-type="fig" rid="fig1">Figure 1B</xref>), and on 31% of the days 50 or more volunteers were active. This illustrates the importance of engaging with the volunteers via multiple channels to ensure an active and sustainable project. Overall, this is a mean of 84.1 classifications per volunteer; however, this hides a large amount of variation in the number of classifications done by individual volunteers. Almost half of the volunteers (4299) did ten or fewer classifications and 1115 classified only a single image whilst the ten volunteers who participated the most did 103,569 classifications between them which is 13.1% of the total. The Gini-coefficient is a useful way to measure these unequal levels of participation, and for this dataset it is 0.85 (<xref ref-type="fig" rid="fig1">Figure 1C</xref>).</p></sec><sec id="s2-3"><title>Comparison to other Zooniverse projects</title><p>The activity within the first 100 days of launch has been used to benchmark and compare different Zooniverse projects from several academic disciplines (<xref ref-type="bibr" rid="bib21">Spiers et al., 2020</xref>). A total of 381,964 classifications were done in the first hundred days after launch by a total of 6237 users of which 3733 were registered and so were unique. Several Zooniverse projects have attracted many more users and classifications; however, these are all ecology or astronomy projects which are the mainstay of the Zooniverse.</p><p>Since the number of classifications is heavily influenced by the difficulty of the task, it can be more illuminating to compare the Gini coefficients of different projects. <xref ref-type="bibr" rid="bib1">Cox et al., 2015</xref> measured a mean Gini coefficient across several Zooniverse projects of 0.81, whilst a later and more comprehensive study <xref ref-type="bibr" rid="bib20">Spiers et al., 2019</xref> demonstrated that Zooniverse projects had Gini coefficients in the range 0.54–0.94 with a mean of 0.80. They also suggested that biomedical projects had lower Gini coefficients, with a mean Gini coefficient of 0.67; however, this was only based on three projects. BashTheBug attracted more users, completed more classifications and had a higher Gini coefficient than any of these three biomedical projects (<xref ref-type="bibr" rid="bib20">Spiers et al., 2019</xref>). A more recent biomedical project, Etch-a-cell, that launched at a similar time to BashTheBug had a Gini coefficient of 0.83 (<xref ref-type="bibr" rid="bib21">Spiers et al., 2020</xref>). BashTheBug therefore has a higher than average level of participation inequality, having the 17th highest Gini coefficient out of 63 Zooniverse projects surveyed (<xref ref-type="bibr" rid="bib20">Spiers et al., 2019</xref>).</p></sec><sec id="s2-4"><title>Time spent</title><p>The time spent by a volunteer classifying a single drug image varied from a few seconds up to hours; the latter are assumed to be data artefacts, for example when a volunteer leaves a browser window open. The distribution of time spent per image shows no appreciable differences when calculated as function of the incubation time with a mode of 3.5 s (<xref ref-type="fig" rid="fig1">Figure 1D</xref>, <xref ref-type="fig" rid="fig1s2">Figure 1—figure supplement 2</xref>), which is unexpected given after only 7 days of incubation there is little or no bacterial growth. After 14 days incubation there are, however, observable differences between how long the volunteers spent classifying each drug (<xref ref-type="fig" rid="fig1s3">Figure 1—figure supplement 3</xref>).</p></sec><sec id="s2-5"><title>Classification validity</title><p>The tutorial on the Zooniverse website (<xref ref-type="fig" rid="fig1s4">Figure 1—figure supplement 4</xref>) encouraged volunteers to check that the control wells both contain bacterial growth – if not then the drug image should be marked as having ‘No Growth in either of the ‘No Antibiotic’ wells’. They were also asked to check if any of the drug wells contain growth very different to all the others (contamination), inconsistent growth (skip wells), or anything else that would prevent a measurement being taken (artefacts). If any of these are true, they were asked to mark the drug image as “Cannot classify”. In the analysis these were aggregated into a single dilution (NR – not read). In all cases, if a simple majority make a classification of NR, then this is always returned as the result. All NR results are excluded from calculations of the exact or essential agreement.</p></sec><sec id="s2-6"><title>Expert measurements</title><p>Each drug image was also measured by a laboratory scientist using a Vizion instrument as well as programmatically by some software, AMyGDA (<xref ref-type="bibr" rid="bib3">Fowler, 2017</xref>). Although the AMyGDA software is reproducible, it will often classify artefacts, such as air bubbles, contamination, shadows and sediment, as bacterial growth and is also likely to assess a well as containing no bacterial growth when the level of growth is very low. By contrast, laboratory scientists are less consistent but can recognise and ignore artefacts. Since the sources of error for each these methods are different, we constructed a consensus dataset with an assumed reduced error rate by only including readings where both of these independent methods agree on the value of the MIC. We will refer to this as the ‘Expert +AMyGDA’ dataset and the larger dataset simply consisting of all the readings taken by the laboratory scientist as the ‘Expert’ dataset. We shall further assume the error-rate in the ‘Expert +AMyGDA’ consensus dataset is negligible, allowing us to use it as a reference dataset, which in turn will allow us to infer the performance of the volunteers by comparison.</p><p>A total of 12,488 drug images were read after 14 days incubation (<xref ref-type="supplementary-material" rid="supp1">Supplementary file 1</xref>); for 6205 (49.7%) of these both the laboratory scientist (Expert) and the software (AMyGDA) returned the same MIC. Since a laboratory scientist would be reasonably expected to make an error ≤ 5% of the time, the majority of the drug images excluded are most likely due to AMyGDA incorrectly reading a drug image with only a minority being genuine errors.</p><p>By constructing the consensus Expert+AMyGDA dataset we are likely to have introduced bias by unwittingly selecting for samples which are easier to read. One possibility is that we may have selected samples with higher than average levels of bacterial growth. We can show that this is not the case since not only is the average level of growth in the positive controls (as measured by AMyGDA) for the Expert+AMyGDA dataset (30.4%) similar to that observed (30.6%) for the larger Expert dataset (<xref ref-type="fig" rid="fig6s3">Figure 6—figure supplement 3</xref>), but the distributions themselves are very similar.</p><p>A second possibility is that drug images with specific growth configurations (for example either no growth or growth in all the wells) are easier to read than drug images where the growth halts. This would imply that the probability of the Expert+AMyGDA measurements agreeing is a function of the dilution MIC, which indeed is what we find (<xref ref-type="supplementary-material" rid="supp1">Supplementary file 1d</xref>). The agreement is highest when there is no growth in any of the drug wells, which makes sense as that is a relatively trivial classification to make. The next highest value is when the dilution is 8, which since 7 of the 14 drugs on the plate have 7 drug wells (Figure 6), corresponds to growth in all 7 drug wells, which is also an easy classification.</p><p>The net effect of this is that the Expert+AMyGDA dataset has a different distribution of measured MICs, including a greater proportion of drug images with a low MIC dilution (61.4% after 14 days incubation have a dilution of 1 or 2, <xref ref-type="fig" rid="fig6s4">Figure 6—figure supplement 4</xref>) compared to the parent Expert dataset (45.8%). One should bear in mind this bias when interpreting the results, and to assist we will consider if key results change when we use the Expert-only dataset.</p></sec><sec id="s2-7"><title>How to compare?</title><p>Ideally one would apply an international standard for antibiotic susceptibility testing (AST) for Mycobacteria which would permit us to assess if a consensus measurement obtained from a crowd of volunteers is sufficiently reproducible and accurate to be accredited as an AST device. Unfortunately, there is no international standard for Mycobacterial AST – the need to subject Mycobacteria to the same processes and standards as other bacteria has been argued elsewhere (<xref ref-type="bibr" rid="bib19">Schön et al., 2020</xref>) – we shall therefore tentatively apply the international standard for aerobic bacteria (<xref ref-type="bibr" rid="bib9">International Standards Organization, 2007</xref>) which requires the results of the proposed antibiotic susceptibility testing method to an appropriate reference method.</p><p>Neither of the measurement methods used in constructing our reference consensus dataset has been endorsed, although broth microdilution using Middlebrook 7H9 media was recently selected by EUCAST as a reference method for determining <italic>M. tuberculosis</italic> MICs (<xref ref-type="bibr" rid="bib19">Schön et al., 2020</xref>) but only for manually-prepared 96-well plates, permitting much larger numbers of wells for each drug. Nor has any software-based approach for reading MICs from 96-well microdilution plates been endorsed by EUCAST, the CLSI or any other international body. Despite this, and in lieu of any other reasonable approach, we shall treat the consensus MICs (the Expert +AMyGDA datset) as a reference dataset and apply ISO 20776–2 (<xref ref-type="bibr" rid="bib9">International Standards Organization, 2007</xref>).</p><p>This requires a new AST method that measures MICs to be compared to the reference method using two key metrics: the <italic>exact agreement</italic> and the <italic>essential agreement</italic>. The former is simply the proportion of definite readings which agree, whilst the latter allows for some variability and is defined as the “MIC result obtained by the AST device that is within plus or minus doubling dilution step from the MIC value established by the reference method” (<xref ref-type="bibr" rid="bib9">International Standards Organization, 2007</xref>). To meet the standard any new AST method must be ≥95% reproducible and ≥90% accurate (both assessed using essential agreement) compared to the reference method (<xref ref-type="bibr" rid="bib9">International Standards Organization, 2007</xref>).</p></sec><sec id="s2-8"><title>Variability in classifications</title><p>Inevitably there is a large degree of variation in the classifications made by different volunteers of the same drug image. Examining all 112,163 classifications made by the volunteers on the 6205 drug images taken after 14 days incubation and comparing them to the consensus of the laboratory scientist and AMyGDA shows that a single volunteer is likely to exactly agree with the Expert +AMyGDA dataset 74.6% of the time, excluding cases where either concluded the drug image could not be read (<xref ref-type="supplementary-material" rid="supp1">Supplementary file 1c</xref>). This rises to 86.4% when only considering essential agreement.</p><p>The magnitude of agreement varies depending on the measured dilution: if the consensus view is that a drug image contains no growth, a single volunteer is likely to agree 64.1% of the time (<xref ref-type="fig" rid="fig2">Figure 2A</xref>), however this falls to 47.8% if the consensus dataset indicates that the first four wells contain growth before rising to 94.5% when the laboratory scientist decides the dilution is 8. We hence recapitulate our earlier observation that drug images with no growth or growth in all wells are easier to read than drug images where only a subset of drug wells contain growth.</p><fig-group><fig id="fig2" position="float"><label>Figure 2.</label><caption><title>Heatmap showing how all the individual BashTheBug classifications (n=214,164) compare to the dilution measured by the laboratory scientist using the Thermo Fisher Vizion instrument after 14 days incubation (n=12,488).</title><p>(<bold>A</bold>) The probability that a single volunteer exactly agrees with the Expert +AMyGDA dataset varies with the dilution. (<bold>B</bold>) The distribution of all dilutions in the Expert +AMyGDA dataset after 14 days incubation. The differences are due to different drugs having different numbers of wells as well as the varying levels of resistance in the supplied strains. NR includes both plates that could not be read due to issues with the control wells and problems with individual drugs such as skip wells. (<bold>C</bold>) The distribution of all dilutions measured by the BashTheBug volunteers. (<bold>D</bold>) A heatmap showing the concordance between the Expert +AMyGDA dataset and the classifications made by individual BashTheBug volunteers. Only cells with <italic>gt</italic><sub>0.1</sub>% are labelled. (<bold>E</bold>) Two example drug images where both the Expert and AMyGDA assessed the MIC as being a dilution of 5 whilst a single volunteer decided no growth could be seen in the image. (<bold>F</bold>) Two example drug images where both the laboratory scientist and a volunteer agreed that the MIC was a dilution of 5. (<bold>G</bold>) Two example drug images where the laboratory scientist decided there was no growth in any of the wells, whilst a single volunteer decided there was growth in the first four wells.</p></caption><graphic mimetype="image" mime-subtype="jpeg" xlink:href="elife-75046.xml.media/fig2.jpg"/></fig><fig id="fig2s1" position="float" specific-use="child-fig"><label>Figure 2—figure supplement 1.</label><caption><title>Heatmap showing how all the individual BashTheBug classifications (n=214,164) compare to the set of dilutions where the measurement made by the laboratory scientist using the Thermo Fisher Vizion instrument and a mirrored box after 14 days incubation concur (n=9402) (<bold>A</bold>).</title><p>The probability that a single volunteer exactly agrees with the Expert dataset varies with the dilution. The distribution of all MIC dilutions after 14 days incubation read by (<bold>B</bold>) laboratory scientists and (<bold>C</bold>) BashTheBug volunteers. NR includes both plates that could not be read due to issues with the control wells and problems with individual drugs such as skip wells. (<bold>D</bold>) A heatmap showing how for each set of images assessed by the laboratory scientist has having a specific dilution as the MIC, the classifications made by BashTheBug volunteers varied considerably. It is normalised so that each row sums to 100% and only cells with >0.1 % are labelled.</p></caption><graphic mimetype="image" mime-subtype="jpeg" xlink:href="elife-75046.xml.media/fig2-figsupp1.jpg"/></fig></fig-group><p>The BashTheBug volunteers are likely to return a higher dilution than the Expert +AMyGDA consensus; this can be seen in the greater proportion of MICs with higher dilutions (<xref ref-type="fig" rid="fig2">Figure 2B–D</xref>). For example a single volunteer is at least 5 ×, and often > 10 ×, more likely to return an MIC one greater than the reference rather than an MIC one lower than the reference (<xref ref-type="fig" rid="fig2">Figure 2D</xref>). We shall return to this bias later.</p><p>When the classification made by an individual volunteer does not agree with the consensus this is often (but not always) because they have misclassified the drug image (<xref ref-type="fig" rid="fig2">Figure 2E–G</xref>). Comparing the classifications made by individual volunteers with the larger, but presumably less accurate, Expert dataset we see that an individual volunteer is less likely to agree with a single laboratory scientist with the overall level of exact agreement falling from 74.6% to 65.3% (<xref ref-type="supplementary-material" rid="supp1">Supplementary file 1</xref>, <xref ref-type="fig" rid="fig2s1">Figure 2—figure supplement 1</xref>). Regardless of the comparison dataset used, it is clear that to achieve satisfactory levels of reproducibility and accuracy one must clearly ask <italic>several</italic> volunteers to assess each drug image and then form a consensus measurement which can be compared to the reference measurement. How should we form that consensus?</p></sec><sec id="s2-9"><title>Consensus</title><p>There are a range of methods one can use to extract a consensus from a set of classifications; the simplest being majority voting, however, this is not practical since an outright majority is not guaranteed. Alternatively one may take the mode, mean or median of the classifications, although the former is not always defined and the last two do not always yield an integer. More sophisticated methods, such as the weighted-majority algorithm (<xref ref-type="bibr" rid="bib12">Littlestone and Warmuth, 1989</xref>), give weights to the classifiers based on their accuracy; however, this requires each volunteer to first classify a ground-truth dataset, which was not available at the start of the project. Given the high level of inequality in participation (<xref ref-type="fig" rid="fig1">Figure 1C</xref>), such methods would be very difficult to apply in practice in our case. We shall therefore limit ourselves here to considering only the mean, median and mode. Since these methods all require the classifications to be numerical, we excluded all readings where the Expert +AMyGDA measurement and/or half or over of the volunteers decided the drug image could not be read. If the classification distribution was bi-modal, then the lower value of the dilution is returned. If necessary, the mean or median were also rounded down.</p></sec><sec id="s2-10"><title>Reproducibility</title><p>To create two consensus measurements by the volunteers of each drug image, two separate sets of 17 classifications were drawn with replacement. By applying the relevant method (mean, median or mode) a consensus dilution was arrived at for each set and then the two results compared. To begin with only drug images with 17 or more classifications were considered and this bootstrapping process was repeated ten times for each drug image in the Expert+AMyGDA dataset. Considering only those drug images taken after 14 days incubation (<xref ref-type="fig" rid="fig3">Figure 3A</xref>, <xref ref-type="supplementary-material" rid="supp1">Supplementary file 1e</xref>), they are more likely to exactly agree with one another when the mode was applied (89.2 ± 0.1%) than the median (86.9 ± 0.1%) or mean (74.3 ± 0.1%). For the essential agreement, we find that the mean now performs best (97.2 ± 0.1%), followed by the median (94.2 ± 0.1%) and mode (94.1 ± 0.1%). Here and throughout, the standard error of the mean is quoted. As one might expect, the reproducibility for MICs measured by two laboratory scientists, as quantified by essential agreement, was previously shown to be slightly higher, at 97.9 % (<xref ref-type="bibr" rid="bib17">Rancoita et al., 2018</xref>).</p><fig-group><fig id="fig3" position="float"><label>Figure 3.</label><caption><title>Taking the mean of 17 classifications is ≥95% reproducible whilst applying either the median or mode is ≥90% accurate.</title><p>(<bold>A</bold>) Only calculating the mean of 17 classifications achieves an essential agreement ≥95% for reproducibility <xref ref-type="bibr" rid="bib9">International Standards Organization, 2007</xref>, followed by the median and the mode. (<bold>B</bold>) Heatmaps of the consensus formed via the mean, median or mode after 14 days incubation. Only drug images from the Expert + AMyGDA dataset are included. (<bold>C</bold>) The essential agreement between a consensus dilution formed from 17 classifications using the median or mode and the consensus Expert +AMyGDA dilution both exceed the required 90% threshold <xref ref-type="bibr" rid="bib9">International Standards Organization, 2007</xref>. (<bold>D</bold>) The heatmaps clearly show how the volunteer consensus dilution is likely to be the same or greater than the Expert + AMyGDA consensus.</p></caption><graphic mimetype="image" mime-subtype="jpeg" xlink:href="elife-75046.xml.media/fig3.jpg"/></fig><fig id="fig3s1" position="float" specific-use="child-fig"><label>Figure 3—figure supplement 1.</label><caption><title>Taking the mean of 17 classifications is ≥95% reproducible whilst none of the methods reach have an essential agreement for accuracy of ≥90% when using the Expert dataset.</title><p>(<bold>A</bold>) Only calculating the mean of 17 classifications achieves an essential agreement ≥95% for reproducibility <xref ref-type="bibr" rid="bib9">International Standards Organization, 2007</xref>, followed by the median and then the mode. There is no specified threshold for exact agreement; the trend is reversed with the mode performing best, followed by the median and then the mean. (<bold>B</bold>) Heatmaps of the consensus formed via the mean, median, or mode after 14 days incubation. Each consensus dilution is a different selection, with replacement, of the original classifications. Drug images from the larger Expert dataset are included. (<bold>C</bold>) The essential agreement between a consensus dilution formed from 17 classifications using the median or mode and the consensus Expert dilution is ≥ 90%, which is the required threshold <xref ref-type="bibr" rid="bib9">International Standards Organization, 2007</xref>. (<bold>D</bold>) The heatmaps clearly show how the volunteer consensus dilution is likely to be the same or greater than the Expert consensus.</p></caption><graphic mimetype="image" mime-subtype="jpeg" xlink:href="elife-75046.xml.media/fig3-figsupp1.jpg"/></fig></fig-group><p>Hence only the mean exceeds the threshold for reproducibility (<xref ref-type="bibr" rid="bib9">International Standards Organization, 2007</xref>) when 17 classifications are used to build a consensus. Repeating the analysis for the drug images in the larger Expert dataset yields the same conclusion (<xref ref-type="fig" rid="fig3s1">Figure 3—figure supplement 1</xref>). The heatmaps (<xref ref-type="fig" rid="fig3">Figure 3B</xref>) show how two consensus measurements arrived at via the mean tend to be similar but not necessarily identical to one another, whilst two consensus measurements derived using the mode are more likely to agree with one another but also are more likely to arrive at very different values. The median sits in between these two extremes.</p></sec><sec id="s2-11"><title>Accuracy</title><p>Comparing the consensus measurements from the volunteers to the set of MICs in the Expert+AMyGDA dataset yields a different picture (<xref ref-type="fig" rid="fig3">Figure 3C</xref>). The mode exactly agrees with the reference 80.9 ± 0.1% of the time, followed by the median (78.1 ± 0.1%) and then mean (68.4 ± 0.1%). The mean, despite performing best for reproducibility, has the lowest level of essential agreement (as well as exact agreement) with the Expert+AMyGDA readings (88.5 ± 0.1%), with the median (90.2 ± 0.1%) and mode (91.0 ± 0.1%) both exceeding the 90% accuracy threshold (<xref ref-type="bibr" rid="bib9">International Standards Organization, 2007</xref>).</p><p>The heatmaps show how the consensus dilution of the classifications made by the volunteers is much more likely to be higher than the Expert+AMyGDA measurement than lower (<xref ref-type="fig" rid="fig3">Figure 3D</xref>), regardless of the consensus method, indicating perhaps that volunteers are more likely than laboratory scientists to classify any dark regions in a well as bacterial growth, or that laboratory scientists are more willing to discount some features as artefacts for example air bubbles or sediment. Repeating the analysis using the Expert dataset (<xref ref-type="fig" rid="fig3s1">Figure 3—figure supplement 1</xref>) leads to lower values for the exact and essential agreements for all consensus methods – this is to be expected since this the Expert reference dataset contains a larger proportion of errors than the Expert+AMyGDA dataset.</p></sec><sec id="s2-12"><title>Which method to choose?</title><p>Despite being the most reproducible method as measured by essential agreement, we discount the mean since it suffers from relatively poor levels of exact agreement for both reproducibility and accuracy and its performance falls faster than the other methods when <inline-formula><mml:math id="inf1"><mml:mi>n</mml:mi></mml:math></inline-formula> is decreased. The median and mode have very similar reproducibilities and accuracy and we conclude they perform equally well. We can infer from this that bi-modal classification distributions are rare and that the median is often identical to the mode.</p></sec><sec id="s2-13"><title>Reducing the number of classifications</title><p>Clearly it would be desirable and ethical to only require the volunteers to complete the minimum number of classifications to achieve an acceptable result. The simplest way to do this is to decrease the number of classifications, <inline-formula><mml:math id="inf2"><mml:mi>n</mml:mi></mml:math></inline-formula>, before a drug image is retired – this reduces both the reproducibility and accuracy of the consensus measurements (<xref ref-type="fig" rid="fig4">Figure 4</xref>, <xref ref-type="supplementary-material" rid="supp1">Supplementary file 1e, f</xref>), however perhaps not by as much as one might expect. The mean exceeds the essential agreement ≥ 95% reproducibility threshold for <inline-formula><mml:math id="inf3"><mml:mrow><mml:mi>n</mml:mi><mml:mo>≥</mml:mo><mml:mn>13</mml:mn></mml:mrow></mml:math></inline-formula>, whilst the mode and the median satisfy the accuracy criterion of essential agreement ≥ 90% for <inline-formula><mml:math id="inf4"><mml:mrow><mml:mi>n</mml:mi><mml:mo>≥</mml:mo><mml:mn>3</mml:mn></mml:mrow></mml:math></inline-formula> and <inline-formula><mml:math id="inf5"><mml:mrow><mml:mi>n</mml:mi><mml:mo>≥</mml:mo><mml:mn>11</mml:mn></mml:mrow></mml:math></inline-formula>, respectively (<xref ref-type="fig" rid="fig4">Figure 4</xref>, <xref ref-type="supplementary-material" rid="supp1">Supplementary file 1f</xref>). Similar trends are observed when the Expert dataset is used as the reference (<xref ref-type="fig" rid="fig4s1">Figure 4—figure supplement 1</xref>) or when the plates are incubated for different periods of time (<xref ref-type="fig" rid="fig4s2">Figure 4—figure supplements 2</xref> and <xref ref-type="fig" rid="fig4s3">3</xref>). Accuracy is hence less sensitive than reproducibility to reducing the number of classifications used to build the consensus and, depending on the consensus method used, the number of classifications can be reduced whilst still maintaining acceptable levels of accuracy.</p><fig-group><fig id="fig4" position="float"><label>Figure 4.</label><caption><title>Reducing the number of classifications, <inline-formula><mml:math id="inf6"><mml:mi>n</mml:mi></mml:math></inline-formula>, used to build the consensus dilution decreases the reproducibility and accuracy of the consensus measurement.</title><p>(<bold>A</bold>) The consensus dilution becomes less reproducible as the number of classifications is reduced, as measured by both the exact and essential agreements. (<bold>B</bold>) Likewise, the consensus dilution becomes less accurate as the number of classifications is decreased, however the highest level of exact agreement using the mean is obtained when <inline-formula><mml:math id="inf7"><mml:mrow><mml:mi>n</mml:mi><mml:mo>=</mml:mo><mml:mn>3</mml:mn></mml:mrow></mml:math></inline-formula> and the mode, and to a lesser extent the median, are relatively insensitive to the number of classifications. These data are all with respect to the Expert +AMyGDA dataset.</p></caption><graphic mimetype="image" mime-subtype="jpeg" xlink:href="elife-75046.xml.media/fig4.jpg"/></fig><fig id="fig4s1" position="float" specific-use="child-fig"><label>Figure 4—figure supplement 1.</label><caption><title>Reducing the number of classifications, <inline-formula><mml:math id="inf8"><mml:mi mathsize="90%">n</mml:mi></mml:math></inline-formula>, used to build the consensus dilution decreases the reproducibility and accuracy of the consensus measurement.</title><p>(<bold>A</bold>) The consensus dilution becomes less reproducible as the number of classifications is reduced, as measured by both the exact and essential agreements. (<bold>B</bold>) Likewise, the consensus dilution becomes less accurate as the number of classifications is decreased, however the highest level of exact agreement using the mean is obtained when <inline-formula><mml:math id="inf9"><mml:mrow><mml:mi mathsize="90%">n</mml:mi><mml:mo mathsize="90%" stretchy="false">=</mml:mo><mml:mn mathsize="90%">3</mml:mn></mml:mrow></mml:math></inline-formula> and the mode, and to a lesser extent the median, are relatively insensitive to the number of classifications. These data are all with respect to the Expert dataset.</p></caption><graphic mimetype="image" mime-subtype="jpeg" xlink:href="elife-75046.xml.media/fig4-figsupp1.jpg"/></fig><fig id="fig4s2" position="float" specific-use="child-fig"><label>Figure 4—figure supplement 2.</label><caption><title>Altering the number of days incubation does not markedly affect the observed trends in reproducibility.</title><p>Shown are results for the Expert +AMyGDA dataset after (<bold>A</bold>) 7, (<bold>B</bold>) 10, (<bold>C</bold>) 14 and (<bold>D</bold>) 21 days of incubation. A previous study (<xref ref-type="bibr" rid="bib17">Rancoita et al., 2018</xref>) showed that optimal results were achieved after 14 days incubation.</p></caption><graphic mimetype="image" mime-subtype="jpeg" xlink:href="elife-75046.xml.media/fig4-figsupp2.jpg"/></fig><fig id="fig4s3" position="float" specific-use="child-fig"><label>Figure 4—figure supplement 3.</label><caption><title>Altering the number of days incubation does not markedly affect the observed trends in accuracy.</title><p>Shown are results for the Expert +AMyGDA dataset after (<bold>A</bold>) 7, (<bold>B</bold>) 10, (<bold>C</bold>) 14 and (<bold>D</bold>) 21 days of incubation. A previous study (<xref ref-type="bibr" rid="bib17">Rancoita et al., 2018</xref>) showed that optimal results were achieved after 14 days incubation.</p></caption><graphic mimetype="image" mime-subtype="jpeg" xlink:href="elife-75046.xml.media/fig4-figsupp3.jpg"/></fig><fig id="fig4s4" position="float" specific-use="child-fig"><label>Figure 4—figure supplement 4.</label><caption><title>Segmenting the drug images by the mean amount of growth in the positive control wells (<xref ref-type="fig" rid="fig6s3">Figure 6—figure supplement 3</xref>) does not markedly affect the reproducibility of the three consensus methods.</title><p>The plates are split into those with (<bold>A</bold>) low (≤ 10 %) growth, (<bold>B</bold>) medium (10 < growth ≤) growth and (<bold>C</bold>) high (> 50 %) growth. The drug images from the Expert +AMyGDA dataset were used and the proportion with MIC is the proportion of consensus readings that are a definite numerical minimum inhibitory concentration.</p></caption><graphic mimetype="image" mime-subtype="jpeg" xlink:href="elife-75046.xml.media/fig4-figsupp4.jpg"/></fig><fig id="fig4s5" position="float" specific-use="child-fig"><label>Figure 4—figure supplement 5.</label><caption><title>Segmenting the drug images by the mean amount of growth in the positive control wells (<xref ref-type="fig" rid="fig6s3">Figure 6—figure supplement 3</xref>) does not markedly affect the accuracy of the three consensus methods.</title><p>The plates are split into those with (<bold>A</bold>) low (≤ 10% %) growth, (<bold>B</bold>) medium (10 < growth ≤ 50 %) growth and (<bold>C</bold>) high (> 50 %) growth. The drug images from the Expert +AMyGDA dataset were used and the proportion with MIC is the proportion of consensus readings that are a definite numerical minimum inhibitory concentration.</p></caption><graphic mimetype="image" mime-subtype="jpeg" xlink:href="elife-75046.xml.media/fig4-figsupp5.jpg"/></fig></fig-group></sec><sec id="s2-14"><title>Can we improve matters?</title><p>Retiring all drug images after a fixed number of classifications is simple but does not take account of the relative difficulty of the classification task. If one was able to group the drug images by difficulty, either before upload or dynamically during classification, then one could optimally target the work undertaken by the volunteers. Due to the inherent difficulties in culturing <italic>M. tuberculosis</italic>, there is a broad distribution of growth in the positive control wells after 14 days incubation (<xref ref-type="fig" rid="fig6s3">Figure 6—figure supplement 3</xref>). One might expect that drug images with poor growth would be more challenging to classify, however, segmenting by low, medium and high growth shows the amount of growth in the positive control wells has little effect on either the reproducibility (<xref ref-type="fig" rid="fig4s4">Figure 4—figure supplement 4</xref>) or accuracy (<xref ref-type="fig" rid="fig4s5">Figure 4—figure supplement 5</xref>), regardless of the consensus method and number of classifications employed.</p><p>Alternatively one could use the first few classifications performed by the volunteers to assess the difficulty of each drug image. For example, if the first <inline-formula><mml:math id="inf10"><mml:mi>n</mml:mi></mml:math></inline-formula> volunteers all return the same classification, then it is reasonable to assume that this is a straightforward image and it can be retired, with the remainder accruing additional classifications. Ideally one would want to develop an algorithm that assessed the likelihood of the classification not being altered by more classifications to allow a dynamic decision about when to halt, however, applying such an approach was not possible at the time in the Zooniverse.</p><p>To estimate the potential value in applying a simple approach to dynamically retiring drug images, we shall consider applying the median after 14 days of incubation and will arbitrarily retire a drug image if the first three volunteers all made the same classification, with all other drug images being retired after 17 classifications. This simple protocol reduces the number of classifications required to <inline-formula><mml:math id="inf11"><mml:mrow><mml:mi>n</mml:mi><mml:mo>=</mml:mo><mml:mn>8.8</mml:mn></mml:mrow></mml:math></inline-formula>, a reduction of 48%, and the reproducibility, as measured by exact agreement, rises from 86.8% to 87.6%, whilst the essential agreement remains unchanged (94.2–94.4%). The accuracy, assessed in the same way, behaves similarly with the exact agreement increasing from 78.1% to 78.8% with the essential agreement remaining unaltered (90.2–90.3%). Hence retiring some of the drug images at <inline-formula><mml:math id="inf12"><mml:mrow><mml:mi>n</mml:mi><mml:mo>=</mml:mo><mml:mn>3</mml:mn></mml:mrow></mml:math></inline-formula> not only dramatically reduces the number of classifications required but also improves the result in a few cases, presumably because the subsequent classifications have a small chance of altering the dilution by a single unit, hence worsening the exact agreement but not affecting the essential agreement.</p><p>A fairer test is to ask if this dynamic approach improves performance if we are constrained to a fixed number of total classifications: if we choose <inline-formula><mml:math id="inf13"><mml:mrow><mml:mi>n</mml:mi><mml:mo>=</mml:mo><mml:mn>9</mml:mn></mml:mrow></mml:math></inline-formula>, then the reproducibility of the median (as measured by exact and essential agreements) improves from 83.0–92.2% to 87.6–94.4% and the accuracy, measured in the same way, improves slightly from 77.2–89.6% to 78.8–90.3%. We therefore conclude that even a simple dynamic approach to retiring images would minimise the work done by the volunteers / allow more images to be classified.</p></sec><sec id="s2-15"><title>Variation by drug</title><p>So far we have analysed the reproducibility and accuracy of consensus MICs obtained from a crowd of volunteers, thereby aggregating the results for each of the 13 anti-tuberculars (excl. PAS) present on the UKMYC5 plate design. The reproducibility of each drug, as measured by the exact and essential agreements, varies between 79.6–91.4% and 90.8–97.0%, respectively (<xref ref-type="fig" rid="fig5">Figure 5A</xref>). This is consistent with an earlier study which found that intra- and inter-laboratory reproducibility when the drugs were read by a scientist using a Vizion instrument varied between 93.4-99.3 % and 87.1-99.1 % (as measured by essential agreement, excl. PAS), respectively (<xref ref-type="bibr" rid="bib17">Rancoita et al., 2018</xref>). Earlier we showed that the reproducibility of the whole plate under these conditions when assessed using the essential agreement is 94.2 ± 0.1% (<xref ref-type="fig" rid="fig3">Figure 3A</xref>) – this is below the 95% threshold specified by an international standard for aerobic bacteria (<xref ref-type="bibr" rid="bib9">International Standards Organization, 2007</xref>). Applying the same threshold to each drug we find that five out of the 13 drugs meet or exceed the threshold whilst the plate as a whole does not. The accuracy of each drug varies more widely: between 48.4–88.9% and 82.8–94.7% when assessed using the exact and essential agreement, respectively. Hence whilst the accuracy of the plate as a whole was 90.2 ± 0.1%, just exceeding the 90% threshold, only six out of 13 drugs surpassed the same threshold.</p><fig-group><fig id="fig5" position="float"><label>Figure 5.</label><caption><title>The reproducibility and accuracy of the consensus MICs varies by drug.</title><p>Consensus MICs were arrived at by taking the median of 17 classifications after 14 days incubation. The essential and exact agreements are drawn as red and green bars, respectively. For the former the minimum thresholds required are 95% and 90% for the reproducibility and accuracy, respectively (<xref ref-type="bibr" rid="bib9">International Standards Organization, 2007</xref>). See (<xref ref-type="fig" rid="fig5s1">Figure 5—figure supplement 1</xref>) for the other consensus methods.</p></caption><graphic mimetype="image" mime-subtype="jpeg" xlink:href="elife-75046.xml.media/fig5.jpg"/></fig><fig id="fig5s1" position="float" specific-use="child-fig"><label>Figure 5—figure supplement 1.</label><caption><title>The reproducibility and accuracy after 14 days incubation of the 13 antibiotics on the UKMYC5 plate.</title><p>A total of 17 classifications were used for each measurement and either the mean or mode was used to obtain a consensus reading of the (<bold>A</bold>) reproducibility and (<bold>B</bold>) accuracy. The essential agreement is drawn in red and the required thresholds are 95% and 90% for the reproducibility and accuracy, respectively (<xref ref-type="bibr" rid="bib9">International Standards Organization, 2007</xref>). The exact agreement is drawn in green and no threshold is defined. The drug abbreviations are defined in (<xref ref-type="fig" rid="fig6s1">Figure 6—figure supplement 1</xref>). The dataset used was Expert +AMyGDA.</p></caption><graphic mimetype="image" mime-subtype="jpeg" xlink:href="elife-75046.xml.media/fig5-figsupp1.jpg"/></fig></fig-group><p>The variation in reproducibility and accuracy between anti-tuberculars, as well as between the exact and essential agreement for a single compound, is due to a number of factors, not limited to the number and concentration range of the wells on the plate design, the mechanisms of both action and resistance, the prevalence of resistance and the degradation rate after lyophilisation, both during storage and after inoculation. For example, kanamycin passes the reproducibility and accuracy thresholds we have adopted and this is likely due to there being relatively few (five) drug wells on the UKMYC5 plate (<xref ref-type="fig" rid="fig6s1">Figure 6—figure supplement 1</xref>) and the mechanism of resistance being such a substantial proportion of samples either do not grow in any well, or grow in all the drug wells, making measurement more straightforward. The mechanism of action of each compound is likely to affect how ‘easy’ it is to determine the minimum inhibitory concentration. From the striking differences in exact and essential accuracies of reading ethambutol, moxifloxacin and linezolid we hypothesise the Mycobacterial growth diminishes more gradually with increasing concentration for these drugs, rather than coming to an abrupt end, as it does for other compounds.</p><p>This whole analysis could be considered somewhat moot since the UKMYC5 96-well plate would be treated as a single entity (or medical device) if accreditation were to be sought and therefore the results for individual compounds would likely not be considered. One unintended consequence of the current standards is therefore that one could improve the performance of a plate design by dropping compounds with lower-than-average performance, even if this is clinically not desirable, rather than work to for example improve the performance of the measurement methods.</p></sec></sec><sec id="s3" sec-type="discussion"><title>Discussion</title><p>A crowd of volunteers can reproducibly and accurately measure the growth of a bacterial pathogen on a 96-well broth microdilution plate, thereby demonstrating the potential for clinical microbiology to embrace and combine contrasting measurement methods. No Mycobacterial antibiotic susceptibility testing standard exists, although efforts are underway to establish a reference method (<xref ref-type="bibr" rid="bib18">Schön et al., 2019</xref>), and so we applied the standard for antibiotic susceptibility testing of aerobic bacteria (<xref ref-type="bibr" rid="bib9">International Standards Organization, 2007</xref>). Forming a consensus by applying the mode or median to 17 independent classifications performs better overall than the mean, and the reproducibility of both these methods, as measured by the essential agreement, is 94.2% and 94.1%, respectively (<xref ref-type="fig" rid="fig3">Figure 3</xref>). This is below than the 95% threshold set by ISO for aerobic bacteria and therefore the volunteers do not pass this standard. The accuracy of the crowd, as measured by the essential agreement, is 90.2% and 91.0% when the median and mode, respectively, are applied to produce a consensus measurement – these values are above the required 90% threshold (<xref ref-type="bibr" rid="bib9">International Standards Organization, 2007</xref>), and therefore the volunteers are sufficiently accurate (but not quite reproducible enough) to be classified as an antibiotic susceptibility testing (AST) device.</p><p>The BashTheBug project has therefore been successful in its primary aim, i.e. the consensus readings made by the crowd of volunteers have been successfully used within the CRyPTIC project to reduce the measurement error in the dataset of 268,281 (13 drugs × 20,637 samples) minimum inhibitory concentrations. CRyPTIC has already significantly contributed to the first resistance catalogue published by the World Health Organization (<xref ref-type="bibr" rid="bib26">Walker et al., 2022</xref>), carried out the largest genome-wide association study of <italic>M. tuberculosis</italic> (<xref ref-type="bibr" rid="bib22">The CRyPTIC Consortium, 2021</xref>) and assessed the magnitude of MIC-shifts for the most common mutations that confer resistance (<xref ref-type="bibr" rid="bib22">The CRyPTIC Consortium, 2021</xref>). The intention is to accelerate the shift from culture-based to genetics-based AST, which is already faster and cheaper than the culture-based alternatives for tuberculosis (<xref ref-type="bibr" rid="bib16">Pankhurst et al., 2016</xref>) and this is well underway (<xref ref-type="bibr" rid="bib25">Walker et al., 2017</xref>). Ultimately shifting to a genetics-based paradigm, where the susceptibility of a pathogen to an antibiotic is inferred from the genome of the pathogen, potentially offers a route to making pathogen diagnostics much more widely available. On a more personal note, it has been very fulfilling engaging with the public through the BashTheBug project and we urge other scientists to consider citizen science if it is appropriate for their research. It is our experience that, whilst the sheer volume of classifications a crowd can achieve can be staggering, it requires us, as researchers, to invest time and effort in regularly communicating with the volunteers which we mainly achieved through our blog, Twitter account, the Zooniverse talk boards and newsletter emails.</p><p>The volunteers are fast, taking on average 3.5 s per drug image, and therefore a single plate requires slightly less than 13 min of volunteer time to read if 17 classifications are amassed for each drug. Reducing the number of classifications before an image is retired reduces the reproducibility and accuracy, but not by as much as one might expect. A more nuanced approach would be to retire a drug image early if the first few classifications are identical, however it was not possible to define this type of dynamical rule in the Zooniverse portal when these data were collected. The level of participation by the volunteers was very unequal with a small cadre of volunteers doing very large numbers with ten volunteers doing, on average, over 10,000 classifications each which is more than many of the laboratory scientists who are considered the experts!</p><p>Compared to the measurements taken by the laboratory scientists, the consensus dilution arrived at by the volunteers tends to be higher, indicating a bias to overcall (<xref ref-type="fig" rid="fig3s1">Figure 3—figure supplement 1</xref>), which is supported by anecdotal observations of people classifying drug images at public engagement events where they often choose a higher dilution ‘to be on the safe side’. As volunteers do more classifications they become more experienced and, on the whole, the magnitude of the bias reduces but does not disappear (<xref ref-type="fig" rid="fig6s5">Figure 6—figure supplement 5</xref>). By contrast, the AMyGDA computer software has been noted to have the opposite bias – that is is more likely to undercall compared to the expert (<xref ref-type="bibr" rid="bib6">Fowler et al., 2018c</xref>). At present in the CRyPTIC project these biases are ignored, that is if the AMyGDA MIC and the consensus reached by BashTheBug agree it is assumed that is a correct measurement. A more nuanced approach that took into account these systematic biases would, we expect, reduce the levels of measurement error in the CRyPTIC MIC dataset yet further and slightly increase the number of MICs where two or more measurement methods agree with one another (<xref ref-type="bibr" rid="bib2">CRyPTIC Consortium, 2022</xref>).</p><p>The reproducibility and accuracy of any method used to read a 96-well microtitre plate, whether that is laboratory scientists using a Thermo Fisher Vizion instrument or citizen scientists visually examining drug images, depends on a range of factors from the prevalence of drug-resistant samples in the dataset to which drugs are included in the plate design and the number and concentrations of their allotted wells. For the UKMYC5 plate design, both the Expert and either the AMyGDA or BashTheBug measurements are more likely to agree with one another at low dilutions where there is little or no <italic>M. tuberculosis</italic> growth in the drug wells (<xref ref-type="supplementary-material" rid="supp1">Supplementary file 1d, i</xref>), hence reducing the number of resistant samples would artificially ‘improve’ performance yet the standards do not specify the degree of resistance in any test dataset (<xref ref-type="bibr" rid="bib9">International Standards Organization, 2007</xref>). The requirement to have quality control strains that have definite growth in all the drug wells unintentionally mitigates against this risk. The dataset used here was based on 19 external quality assessment strains and therefore whilst it included some degree of resistance for all 13 drugs, there was only a single strain resistant to clofazimine, bedaquiline or delamanid and no strain was resistant to linezolid (<xref ref-type="bibr" rid="bib17">Rancoita et al., 2018</xref>). For isoniazid and rifampicin, eight and seven of the 19 EQA strains, respectively were resistant and hence the prevalence of resistance for these drugs is much greater than would be expected to be encountered in most countries. Clearly studies including a much more diverse range of strains, for example clinical isolates, would be more definitive. Since the 13 antituberculars on the UKMYC5 plate (<xref ref-type="fig" rid="fig6s1">Figure 6—figure supplement 1</xref>) also all perform differently (<xref ref-type="fig" rid="fig5">Figure 5</xref>) different plate designs will perform differently, which is important as it is the plate that would be accredited, rather than the individual compounds.</p><p>Although the primary aim of this study was to assess whether the measurements produced by a crowd of volunteers are sufficiently reproducible and accurate to help reduce the measurement error in datasets containing large numbers of microtitre plates (as is being collected by the CRyPTIC project and others) the resulting dataset of classifications is ideally suited to train machine-learning models. This is increasingly recognised as an important use of citizen science (<xref ref-type="bibr" rid="bib24">Trouille et al., 2019</xref>) and one could envisage training a light-weight machine-learning algorithm able to run on a mobile device which, by taking a photograph of a 96-well plate, could automatically read the minimum inhibitory concentrations. The best use of such a device would likely be to act as a double check for readings taken by the laboratory scientist. Alternatively, one could build a hybrid approach where e.g. small crowds of experts could examine plates used in a clinical microbiology service – these could be particularly difficult drug images or could be a random sample for quality assurance purposes. This type of hybrid approach would also help with training laboratory scientists which would help reduce the barrier to using 96-well microtitre plates for <italic>M. tuberculosis</italic> AST in clinical microbiology laboratories, especially in low- and middle-income countries. Finally, it is likely that each volunteer has their own individual bias and variability and constructing consensus methods (<xref ref-type="bibr" rid="bib28">Zhu et al., 2014</xref>) that take these into account would likely further improve the performance of crowds of citizen scientists.</p></sec><sec id="s4" sec-type="materials|methods"><title>Materials and methods</title><sec id="s4-1"><title>Plate design</title><p>The CRyPTIC project has collected a large number (20,637) of clinical TB samples, each having the MICs of a panel of 13 antibiotics measured using a 96-well broth microdilution plate with the majority of samples also undergoing whole genome sequencing. Samples were collected by 14 laboratories based in 11 countries spread across five continents and included all the major lineages and were biased towards drug resistance (<xref ref-type="bibr" rid="bib2">CRyPTIC Consortium, 2022</xref>, <xref ref-type="bibr" rid="bib23">The CRyPTIC Consortium, 2022</xref>). Two plate designs were used, with the second (UKMYC6) being an evolution of the first (UKMYC5) plate design. In this paper we shall focus on data collected using the UKMYC5 plate design which is itself a variant of the MYCOTB 96-well microdilution plate manufactured by Thermo Fisher containing 14 anti-TB drugs. UKMYC5 includes two repurposed compounds (linezolid and clofazimine) and two new compounds (delamanid and bedaquiline). Since 96-well plates have 8 rows and 12 columns, fitting 14 drugs, alongside two positive control wells, onto the plate necessitated a complex design (<xref ref-type="fig" rid="fig6s1">Figure 6—figure supplement 1</xref>).</p><p>Each of the antibiotics on the UKMYC5 plate is present at 5–8 concentrations, each double the previous. The smallest concentration that prevents growth of <italic>M. tuberculosis</italic> after two weeks of incubation is the minimum inhibitory concentration (MIC). Since <italic>M. tuberculosis</italic> is notoriously difficult to culture and inspect, relying on a reading taken by a single expert (laboratory scientist) would likely have led to high levels of errors in the dataset.</p></sec><sec id="s4-2"><title>Image dataset</title><p>The images considered by the crowd of volunteers originated from an earlier study whose aim was to assess if the UKMYC5 plate was sufficiently reproducible and accurate for use as a high-throughput research MIC measurement device (<xref ref-type="bibr" rid="bib17">Rancoita et al., 2018</xref>). Seven CRyPTIC laboratories were each sent thirty one vials containing one of 19 external quality assessment (EQA) <italic>M. tuberculosis</italic> strains, including the reference strain H37Rv ATCC 27294 (<xref ref-type="bibr" rid="bib11">Kubica et al., 1972</xref>). The EQA strains varied in their resistance profiles (<xref ref-type="bibr" rid="bib17">Rancoita et al., 2018</xref>). Thirty of the vials were blinded and the last contained the reference strain. Within the thirty vials, some of EQA strains were duplicated and two of the blinded strains were also the reference strain. Since some labs only received a subset of the 31 vials (<xref ref-type="supplementary-material" rid="supp1">Supplementary file 1a</xref>), a total of 447 plates were inoculated and then incubated for 3 weeks (<xref ref-type="fig" rid="fig6">Figure 6A</xref>). Due to the slow growth rate of <italic>M. tuberculosis</italic>, minimum inhibitory concentrations of the 14 drugs on the plate were measured after 7, 10, 14, and 21 days by two scientists using a Thermo Fisher Sensititre Vizion Digital MIC viewing system, a mirrored-box and a microscope. One or two photographs were also taken each time using the Vizion instrument (<xref ref-type="fig" rid="fig6">Figure 6B</xref>).</p><fig-group><fig id="fig6" position="float"><label>Figure 6.</label><caption><title>Each UKMYC5 plate was read by an Expert, by some software (AMyGDA) and by at least 17 citizen scientist volunteers via the BashTheBug project.</title><p>(<bold>A</bold>) 447 UKMYC5 plates were prepared and read after 7, 10, 14 and 21 days incubation. (<bold>B</bold>) The minimum inhibitory concentrations (MIC) for the 14 drugs on each plate were read by an by Expert, using a Vizion instrument. The Vizion also took a photograph which was subsequently analysed by AMyGDA – this software then composited 14 drug images from each photograph, each containing an image of the two positive control wells. To allow data from different drugs to be aggregated, all MICs were converted to dilutions. (<bold>C</bold>) All drug images were then uploaded to the Zooniverse platform before being shown to volunteers through their web browser. Images were retired once they had been classified by 17 different volunteers. Classification data were downloaded and processed using two Python modules (pyniverse +bashthebug) before consensus measurements being built using different methods.</p></caption><graphic mimetype="image" mime-subtype="jpeg" xlink:href="elife-75046.xml.media/fig6.jpg"/></fig><fig id="fig6s1" position="float" specific-use="child-fig"><label>Figure 6—figure supplement 1.</label><caption><title>The UKMYC5 plate contains 14 different anti-TB drugs.</title><p>A previous study (<xref ref-type="bibr" rid="bib17">Rancoita et al., 2018</xref>) showed that <italic>para</italic>-aminosalicylic acid (PAS) performed poorly and it has been removed from the subsequent UKMYC6 plate design. We have therefore excluded this drug from all analyses. Each drug was contained in 5, 6, 7, or 8 wells with each well having double the concentration of drug as the one before. The concentration of the first and last well in each drug series is labelled (mg/L). Two wells contain no drug and are therefore positive control wells.</p></caption><graphic mimetype="image" mime-subtype="jpeg" xlink:href="elife-75046.xml.media/fig6-figsupp1.jpg"/></fig><fig id="fig6s2" position="float" specific-use="child-fig"><label>Figure 6—figure supplement 2.</label><caption><title>Although the retirement limit within the Zooniverse platform was set to 17, over 1800 images received more classifications than this and a small number were only classified 15 or 16 times.</title></caption><graphic mimetype="image" mime-subtype="jpeg" xlink:href="elife-75046.xml.media/fig6-figsupp2.jpg"/></fig><fig id="fig6s3" position="float" specific-use="child-fig"><label>Figure 6—figure supplement 3.</label><caption><title>The Expert +AMyGDA consensus dataset has the same distribution of bacterial growth in the positive control wells as the Expert dataset after 14 days incubation.</title><p>(<bold>A</bold>) The distribution of the mean positive control well growth, as measured by AMyGDA, for the Expert +AMyGDA dataset. The dataset is arbitrarily split into three categories: low (<10%), medium (10 ≤ growth < 50 %) and high (≥ 50 %) growth. The proportions of the dataset in each category are labelled. (<bold>B</bold>) The distribution of the mean positive control well growth, as measured by AMyGDA, for the Expert dataset. There are around twice as many plates in this dataset (<xref ref-type="supplementary-material" rid="supp1">Supplementary file 1c</xref>).</p></caption><graphic mimetype="image" mime-subtype="jpeg" xlink:href="elife-75046.xml.media/fig6-figsupp3.jpg"/></fig><fig id="fig6s4" position="float" specific-use="child-fig"><label>Figure 6—figure supplement 4.</label><caption><title>The Expert +AMyGDA dataset has a greater proportion of drug images with low dilutions compared to the Expert dataset.</title><p>The growth of the bacteria is also evident as the number of days the sample was incubated for is increased.</p></caption><graphic mimetype="image" mime-subtype="jpeg" xlink:href="elife-75046.xml.media/fig6-figsupp4.jpg"/></fig><fig id="fig6s5" position="float" specific-use="child-fig"><label>Figure 6—figure supplement 5.</label><caption><title>The average bias per volunteer decreases with experience.</title><p>The average bias per volunteer, as defined by the difference between a volunteer’s reading and that from the Expert +AMyGDA dataset, is plotted against the total number of classifications done by each volunteer. Only volunteers who have done 10 or more classifications are plotted.</p></caption><graphic mimetype="image" mime-subtype="jpeg" xlink:href="elife-75046.xml.media/fig6-figsupp5.jpg"/></fig></fig-group><p>The study showed that the UKMYC5 plate is reproducible and that it is optimal to read the plate using either a Thermo Fisher Vizion instrument or a mirrored-box after 14 days of incubation (<xref ref-type="bibr" rid="bib17">Rancoita et al., 2018</xref>). It also showed that <italic>para</italic>-aminosalicylic acid (PAS) was not reproducible on the plate and therefore this drug is excluded in all subsequent analyses. Each image was processed and analysed by some bespoke software, the Automated Mycobacterial Growth Detection Algorithm (AMyGDA), that segmented each photograph, thereby providing a second independent MIC reading of all the drugs on each plate (<xref ref-type="fig" rid="fig6">Figure 6B</xref>; <xref ref-type="bibr" rid="bib6">Fowler et al., 2018c</xref>).</p><p>Early internal tests using the Zooniverse platform showed that asking a volunteer to examine all 96 wells on a plate was too arduous a task. We therefore observed a clinical microbiologist as she examined several photographs of UKMYC5 plates. Rather than considering the <italic>absolute</italic> growth in each well, she was constantly comparing the growth in the wells containing antibiotic back to the positive control wells and therefore was judging what constituted growth <italic>relative</italic> to how well the isolate had grown in the absence of drug. A suitable task is therefore to classify the growth in the wells for a single drug as long as the positive control wells are also provided. The AMyGDA software was therefore modified to composite such <italic>drug images</italic> (<xref ref-type="fig" rid="fig6">Figure 6B</xref>).</p><p>Each UKMYC5 plate yielded 14 composite images, one for each drug. Throughout the following analysis we shall aggregate all the data from the different drugs on the UKMYC5 plate. To facilitate this we shall therefore consider the <italic>dilution</italic>, which is defined as the number of the well in the drug image with the lowest antibiotic concentration which prevents bacterial growth, rather than the minimum inhibitory concentration. Following upload to the Zooniverse platform, the retirement threshold was set to 17 classifications, however some images attracted additional classifications with 421 images having ≥ 34 classifications whilst 89 have 100 or greater (<xref ref-type="supplementary-material" rid="supp1">Supplementary file 1b</xref>, <xref ref-type="fig" rid="fig6s2">Figure 6—figure supplement 2</xref>). This is due, we believe, to images not being removed when retired, but instead having a ”Finished!” sticker added and hence images can accrue additional classifications, especially at times when the volunteers have completed all the uploaded images. Each volunteer is only permitted to classify each image once.</p></sec><sec id="s4-3"><title>Analysis</title><p>The resulting classifications were downloaded from the Zooniverse platform, either by a web browser or using the panoptes-cli command line tool (<xref ref-type="bibr" rid="bib14">McMaster et al., 2021</xref>). Two Python modules were written to parse, store, manipulate and graph this classification data. The first, pyniverse (<xref ref-type="bibr" rid="bib4">Fowler, 2018a</xref>), is designed to be generic for Zooniverse projects whilst the second, bashthebug (<xref ref-type="bibr" rid="bib5">Fowler, 2018b</xref>), inherits from the first and adds functionality specific to BashTheBug (<xref ref-type="fig" rid="fig6">Figure 6C</xref>). Both are freely available to download and use. These Python modules output several Pandas (<xref ref-type="bibr" rid="bib13">McKinney, 2010</xref>) dataframes which were then indexed, filtered and joined to other dataframes containing the sample information and the MIC readings taken by the expert and AMyGDA software. AMyGDA also measured the growth in the two positive control wells and this was also recorded in a dataframe. All subsequent analysis was performed using Python3 in a jupyter-notebook (see Data Availability Statement) and all graphs were plotted using matplotlib.</p></sec><sec id="s4-4"><title>Engagement</title><p>In addition to the Zooniverse project page, which contained background information, a tutorial and FAQs, we setup a blog (<xref ref-type="bibr" rid="bib3">Fowler, 2017</xref>) and a variety of social media channels, focussing mainly on Twitter (@bashthebug). These all used a professionally designed logo and typeface (<xref ref-type="fig" rid="fig6">Figure 6C</xref>), allowing instantaneous recognition of the project, which is important since the Zooniverse platform hosts tens of projects at any one time, and to indirectly convey that this is a professional project and therefore of scientific and societal importance. Since the blog was launched in March 2017 we have written 71 posts, attracting 7,393 visitors who made 13,811 views. At the time of writing, the Twitter account, @bashthebug, has 393 followers and has tweeted 400 times. Finally, the volunteers interacted with one another as well as the project team via the BashTheBug Talk Boards on the Zooniverse platform. A total of 6,255 posts were made by 1042 individuals on 4834 topics. During the course of the project, one of our more experienced volunteers (EMLB) became a de facto moderator by answering so many of the questions posted (>500) which we recognised by giving her moderator status and she is also an author of this paper.</p></sec></sec></body><back><sec sec-type="additional-information" id="s5"><title>Additional information</title><fn-group content-type="competing-interest"><title>Competing interests</title><fn fn-type="COI-statement" id="conf1"><p>No competing interests declared</p></fn><fn fn-type="COI-statement" id="conf2"><p>No competing interests declared</p></fn></fn-group><fn-group content-type="author-contribution"><title>Author contributions</title><fn fn-type="con" id="con1"><p>Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Software, Visualization, Writing – original draft, Writing – review and editing</p></fn><fn fn-type="con" id="con2"><p>Funding acquisition, Writing – review and editing</p></fn><fn fn-type="con" id="con3"><p>Conceptualization, Investigation, Methodology, Writing – review and editing</p></fn><fn fn-type="con" id="con4"><p>Formal analysis, Writing – review and editing</p></fn><fn fn-type="con" id="con5"><p>Funding acquisition, Supervision</p></fn><fn fn-type="con" id="con6"><p>Methodology, Funding acquisition, Project administration, Writing – review and editing</p></fn><fn fn-type="con" id="con7"><p>Conceptualization, Funding acquisition, Writing – review and editing</p></fn><fn fn-type="con" id="con8"><p>Funding acquisition, Writing – review and editing</p></fn><fn fn-type="con" id="con9"><p>Formal analysis, Writing – review and editing</p></fn><fn fn-type="con" id="con10"><p>Investigation, Writing – review and editing</p></fn><fn fn-type="con" id="con11"><p>Investigation, Writing – review and editing</p></fn><fn fn-type="con" id="con12"><p>Conceptualization, Methodology, Software, Writing – review and editing</p></fn><fn fn-type="con" id="con13"><p>Conceptualization, Methodology, Writing – review and editing</p></fn><fn fn-type="con" id="con14"><p>Formal analysis, Writing - original draft, Writing – review and editing</p></fn><fn fn-type="con" id="con15"><p>Writing - original draft, Writing – review and editing</p></fn><fn fn-type="con" id="con16"><p>Conceptualization</p></fn><fn fn-type="con" id="con17"><p>Investigation</p></fn><fn fn-type="con" id="con18"><p>Investigation</p></fn></fn-group></sec><sec sec-type="supplementary-material" id="s6"><title>Additional files</title><supplementary-material id="mdar"><label>MDAR checklist</label><media xlink:href="elife-75046-mdarchecklist1-v3.pdf" mimetype="application" mime-subtype="pdf"/></supplementary-material><supplementary-material id="supp1"><label>Supplementary file 1.</label><caption><title>A supplementary file containing a tables (a-i) is available online.</title><p>The majority of the tables in the supplemental file can also be reproduced using the accompanying jupyter notebook at <ext-link ext-link-type="uri" xlink:href="https://github.com/fowler-lab/bashthebug-consensus-dataset">https://github.com/fowler-lab/bashthebug-consensus-dataset</ext-link>; <xref ref-type="bibr" rid="bib7">Fowler Lab, 2022</xref>.</p></caption><media xlink:href="elife-75046-supp1-v3.tex" mimetype="application" mime-subtype="x-tex"/></supplementary-material></sec><sec sec-type="data-availability" id="s7"><title>Data availability</title><p>The data tables and a Jupyter notebook that allows the user to recreate the majority of figures and tables in both the manuscript and the supplemental information is freely available here: <ext-link ext-link-type="uri" xlink:href="https://github.com/fowler-lab/bashthebug-consensus-dataset">https://github.com/fowler-lab/bashthebug-consensus-dataset</ext-link>, (copy archived at <ext-link ext-link-type="uri" xlink:href="https://archive.softwareheritage.org/swh:1:dir:c34c01c11f511481b3505e897bad80a091e9cf7a;origin=https://github.com/fowler-lab/bashthebug-consensus-dataset;visit=swh:1:snp:473a0168a5c1542beaf77d3a22fa8bdb8c841a41;anchor=swh:1:rev:61ba253936f7ec33a785158ee48c84e63256cd8b">swh:1:rev:61ba253936f7ec33a785158ee48c84e63256cd8b</ext-link>) .It is setup so a user can either clone the repository and run the jupyter-notebook on their local computer (the installation process having installed the pre-requisites) or by clicking the "Launch Binder" button in the README, they can access and run the jupyter-notebook via their web browser, thereby avoiding any installation. I've added a short statement to the manuscript -- please advise if you think it needs changing.</p><p>The following dataset was generated:</p><p><element-citation publication-type="data" specific-use="isSupplementedBy" id="dataset1"><person-group person-group-type="author"><name><surname>Fowler</surname><given-names>PW</given-names></name></person-group><year iso-8601-date="2021">2021</year><data-title>BashTheBug dataset for finding the optimal consensus method</data-title><source>GitHub</source><pub-id pub-id-type="accession" xlink:href="https://github.com/fowler-lab/bashthebug-consensus-dataset">consensus-dataset</pub-id></element-citation></p></sec><ack id="ack"><title>Acknowledgements</title><p>We are very grateful to the Zooniverse volunteer community (Figure 1—figure supplement 1) who contributed their time and energy to this project and to the Zooniverse development team for coding and maintaining the Zooniverse online platform. We thank David Hawkins for designing the BashTheBug logo and typeface and Chris Wood, Oxford Medical Illustration and Dr Nicola Fawcett, <ext-link ext-link-type="uri" xlink:href="https://www.world.com/">livinginamicrobialworld.com</ext-link> for the wild garden of the gut bacteria photographs. For the purpose of open access, the author has applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.</p></ack><ref-list><title>References</title><ref id="bib1"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Cox</surname><given-names>J</given-names></name><name><surname>Oh</surname><given-names>EY</given-names></name><name><surname>Simmons</surname><given-names>B</given-names></name><name><surname>Lintott</surname><given-names>C</given-names></name><name><surname>Masters</surname><given-names>K</given-names></name><name><surname>Greenhill</surname><given-names>A</given-names></name><name><surname>Graham</surname><given-names>G</given-names></name><name><surname>Holmes</surname><given-names>K</given-names></name></person-group><year iso-8601-date="2015">2015</year><article-title>Defining and Measuring Success in Online Citizen Science: A Case Study of Zooniverse Projects</article-title><source>Computing in Science & Engineering</source><volume>17</volume><fpage>28</fpage><lpage>41</lpage><pub-id pub-id-type="doi">10.1109/MCSE.2015.65</pub-id></element-citation></ref><ref id="bib2"><element-citation publication-type="journal"><person-group person-group-type="author"><collab>CRyPTIC Consortium</collab></person-group><year iso-8601-date="2022">2022</year><article-title>Epidemiological cutoff values for a 96-well broth microdilution plate for high-throughput research antibiotic susceptibility testing of <italic>M. tuberculosis</italic></article-title><source>The European Respiratory Journal</source><volume>1949</volume><elocation-id>2200239</elocation-id><pub-id pub-id-type="doi">10.1183/13993003.00239-2022</pub-id><pub-id pub-id-type="pmid">35301246</pub-id></element-citation></ref><ref id="bib3"><element-citation publication-type="web"><person-group person-group-type="author"><name><surname>Fowler</surname><given-names>PW</given-names></name></person-group><year iso-8601-date="2017">2017</year><article-title>Help us fight antibiotic resistance</article-title><ext-link ext-link-type="uri" xlink:href="https://bashthebug.net/bashthebug-on-the-zooniverse/">https://bashthebug.net/bashthebug-on-the-zooniverse/</ext-link><date-in-citation iso-8601-date="2022-06-10">June 10, 2022</date-in-citation></element-citation></ref><ref id="bib4"><element-citation publication-type="software"><person-group person-group-type="author"><name><surname>Fowler</surname><given-names>PW</given-names></name></person-group><year iso-8601-date="2018">2018a</year><data-title>pyniverse: a Python package to analyse classifications made by volunteers in a generic Zooniverse citizen science project</data-title><version designator="ce066f5">ce066f5</version><source>Github</source><ext-link ext-link-type="uri" xlink:href="https://github.com/fowler-lab/pyniverse">https://github.com/fowler-lab/pyniverse</ext-link></element-citation></ref><ref id="bib5"><element-citation publication-type="software"><person-group person-group-type="author"><name><surname>Fowler</surname><given-names>PW</given-names></name></person-group><year iso-8601-date="2018">2018b</year><data-title>bashthebug: a Python package to analyse the results of the Zooniverse volunteers for the BashTheBug citizen science project</data-title><version designator="8b22907">8b22907</version><source>Github</source><ext-link ext-link-type="uri" xlink:href="https://github.com/fowler-lab/bashthebug">https://github.com/fowler-lab/bashthebug</ext-link></element-citation></ref><ref id="bib6"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Fowler</surname><given-names>PW</given-names></name><name><surname>Gibertoni Cruz</surname><given-names>AL</given-names></name><name><surname>Hoosdally</surname><given-names>SJ</given-names></name><name><surname>Jarrett</surname><given-names>L</given-names></name><name><surname>Borroni</surname><given-names>E</given-names></name><name><surname>Chiacchiaretta</surname><given-names>M</given-names></name><name><surname>Rathod</surname><given-names>P</given-names></name><name><surname>Lehmann</surname><given-names>S</given-names></name><name><surname>Molodtsov</surname><given-names>N</given-names></name><name><surname>Walker</surname><given-names>TM</given-names></name><name><surname>Robinson</surname><given-names>E</given-names></name><name><surname>Hoffmann</surname><given-names>H</given-names></name><name><surname>Peto</surname><given-names>TEA</given-names></name><name><surname>Cirillo</surname><given-names>DM</given-names></name><name><surname>Smith</surname><given-names>GE</given-names></name><name><surname>Crook</surname><given-names>DW</given-names></name></person-group><year iso-8601-date="2018">2018c</year><article-title>Automated detection of bacterial growth on 96-well plates for high-throughput drug susceptibility testing of Mycobacterium tuberculosis</article-title><source>Microbiology (Reading, England)</source><volume>164</volume><fpage>1522</fpage><lpage>1530</lpage><pub-id pub-id-type="doi">10.1099/mic.0.000733</pub-id><pub-id pub-id-type="pmid">30351270</pub-id></element-citation></ref><ref id="bib7"><element-citation publication-type="software"><person-group person-group-type="author"><collab>Fowler Lab</collab></person-group><year iso-8601-date="2022">2022</year><data-title>bashthebug-consensus-dataset</data-title><version designator="swh:1:rev:61ba253936f7ec33a785158ee48c84e63256cd8b">swh:1:rev:61ba253936f7ec33a785158ee48c84e63256cd8b</version><source>Software Heritage</source><ext-link ext-link-type="uri" xlink:href="https://archive.softwareheritage.org/swh:1:dir:c34c01c11f511481b3505e897bad80a091e9cf7a;origin=https://github.com/fowler-lab/bashthebug-consensus-dataset;visit=swh:1:snp:473a0168a5c1542beaf77d3a22fa8bdb8c841a41;anchor=swh:1:rev:61ba253936f7ec33a785158ee48c84e63256cd8b">https://archive.softwareheritage.org/swh:1:dir:c34c01c11f511481b3505e897bad80a091e9cf7a;origin=https://github.com/fowler-lab/bashthebug-consensus-dataset;visit=swh:1:snp:473a0168a5c1542beaf77d3a22fa8bdb8c841a41;anchor=swh:1:rev:61ba253936f7ec33a785158ee48c84e63256cd8b</ext-link></element-citation></ref><ref id="bib8"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Galton</surname><given-names>F</given-names></name></person-group><year iso-8601-date="1907">1907</year><article-title>Vox populi</article-title><source>Nature</source><volume>1949</volume><fpage>450</fpage><lpage>451</lpage></element-citation></ref><ref id="bib9"><element-citation publication-type="report"><person-group person-group-type="author"><collab>International Standards Organization</collab></person-group><year iso-8601-date="2007">2007</year><source>ISO 20776-2: Clinical laboratory testing and in vitro diagnostic test systems Technical report</source><publisher-name>International Standards Organization</publisher-name></element-citation></ref><ref id="bib10"><element-citation publication-type="book"><person-group person-group-type="author"><name><surname>Koebel</surname><given-names>J</given-names></name></person-group><year iso-8601-date="1563">1563</year><source>Geometrei von K¨unstlichem Feldmessen</source><publisher-name>Egenolff, Franckfurt am Meyn</publisher-name></element-citation></ref><ref id="bib11"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Kubica</surname><given-names>GP</given-names></name><name><surname>Kim</surname><given-names>TH</given-names></name><name><surname>Dunbar</surname><given-names>FP</given-names></name></person-group><year iso-8601-date="1972">1972</year><article-title>Designation of Strain H37Rv as the Neotype of Mycobacterium tuberculosis</article-title><source>International Journal of Systematic Bacteriology</source><volume>22</volume><fpage>99</fpage><lpage>106</lpage><pub-id pub-id-type="doi">10.1099/00207713-22-2-99</pub-id></element-citation></ref><ref id="bib12"><element-citation publication-type="confproc"><person-group person-group-type="author"><name><surname>Littlestone</surname><given-names>N</given-names></name><name><surname>Warmuth</surname><given-names>M</given-names></name></person-group><year iso-8601-date="1989">1989</year><article-title>In 30th Annual Symposium on Foundations of Computer Science</article-title><conf-name>IEEE</conf-name><fpage>256</fpage><lpage>261</lpage></element-citation></ref><ref id="bib13"><element-citation publication-type="confproc"><person-group person-group-type="author"><name><surname>McKinney</surname><given-names>W</given-names></name></person-group><year iso-8601-date="2010">2010</year><article-title>In Proceedings of the 9th Python in Science Conference</article-title><conf-name>SciPy</conf-name><fpage>51</fpage><lpage>56</lpage></element-citation></ref><ref id="bib14"><element-citation publication-type="software"><person-group person-group-type="author"><name><surname>McMaster</surname><given-names>A</given-names></name><name><surname>Hutchings</surname><given-names>R</given-names></name><name><surname>Allen</surname><given-names>C</given-names></name><name><surname>Wolfenbarger</surname><given-names>Z</given-names></name><name><surname>Dickinson</surname><given-names>H</given-names></name><name><surname>Trouille</surname><given-names>L</given-names></name><name><surname>Johnson</surname><given-names>C</given-names></name></person-group><year iso-8601-date="2021">2021</year><data-title>Panoptes CLI. A command-line interface for Panoptes, the API behind the Zooniverse</data-title><version designator="2.0">2.0</version><source>Github</source><ext-link ext-link-type="uri" xlink:href="https://github.com/zooniverse/panoptes-cli">https://github.com/zooniverse/panoptes-cli</ext-link></element-citation></ref><ref id="bib15"><element-citation publication-type="report"><person-group person-group-type="author"><name><surname>O’Neill</surname><given-names>J</given-names></name></person-group><year iso-8601-date="2016">2016</year><source>Tackling Drug-Resistant Infections Globally: Final Report and Recommendations</source><publisher-name>Technical report</publisher-name></element-citation></ref><ref id="bib16"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Pankhurst</surname><given-names>LJ</given-names></name><name><surname>Del Ojo Elias</surname><given-names>C</given-names></name><name><surname>Votintseva</surname><given-names>AA</given-names></name><name><surname>Walker</surname><given-names>TM</given-names></name><name><surname>Cole</surname><given-names>K</given-names></name><name><surname>Davies</surname><given-names>J</given-names></name><name><surname>Fermont</surname><given-names>JM</given-names></name><name><surname>Gascoyne-Binzi</surname><given-names>DM</given-names></name><name><surname>Kohl</surname><given-names>TA</given-names></name><name><surname>Kong</surname><given-names>C</given-names></name><name><surname>Lemaitre</surname><given-names>N</given-names></name><name><surname>Niemann</surname><given-names>S</given-names></name><name><surname>Paul</surname><given-names>J</given-names></name><name><surname>Rogers</surname><given-names>TR</given-names></name><name><surname>Roycroft</surname><given-names>E</given-names></name><name><surname>Smith</surname><given-names>EG</given-names></name><name><surname>Supply</surname><given-names>P</given-names></name><name><surname>Tang</surname><given-names>P</given-names></name><name><surname>Wilcox</surname><given-names>MH</given-names></name><name><surname>Wordsworth</surname><given-names>S</given-names></name><name><surname>Wyllie</surname><given-names>D</given-names></name><name><surname>Xu</surname><given-names>L</given-names></name><name><surname>Crook</surname><given-names>DW</given-names></name><collab>COMPASS-TB Study Group</collab></person-group><year iso-8601-date="2016">2016</year><article-title>Rapid, comprehensive, and affordable mycobacterial diagnosis with whole-genome sequencing: a prospective study</article-title><source>The Lancet. Respiratory Medicine</source><volume>4</volume><fpage>49</fpage><lpage>58</lpage><pub-id pub-id-type="doi">10.1016/S2213-2600(15)00466-X</pub-id><pub-id pub-id-type="pmid">26669893</pub-id></element-citation></ref><ref id="bib17"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Rancoita</surname><given-names>PMV</given-names></name><name><surname>Cugnata</surname><given-names>F</given-names></name><name><surname>Gibertoni Cruz</surname><given-names>AL</given-names></name><name><surname>Borroni</surname><given-names>E</given-names></name><name><surname>Hoosdally</surname><given-names>SJ</given-names></name><name><surname>Walker</surname><given-names>TM</given-names></name><name><surname>Grazian</surname><given-names>C</given-names></name><name><surname>Davies</surname><given-names>TJ</given-names></name><name><surname>Peto</surname><given-names>TEA</given-names></name><name><surname>Crook</surname><given-names>DW</given-names></name><name><surname>Fowler</surname><given-names>PW</given-names></name><name><surname>Cirillo</surname><given-names>DM</given-names></name><name><surname>Crook</surname><given-names>DW</given-names></name><name><surname>Peto</surname><given-names>TEA</given-names></name><name><surname>Walker</surname><given-names>AS</given-names></name><name><surname>Hoosdally</surname><given-names>SJ</given-names></name><name><surname>Gibertoni Cruz</surname><given-names>AL</given-names></name><name><surname>Grazian</surname><given-names>C</given-names></name><name><surname>Walker</surname><given-names>TM</given-names></name><name><surname>Fowler</surname><given-names>PW</given-names></name><name><surname>Wilson</surname><given-names>D</given-names></name><name><surname>Clifton</surname><given-names>D</given-names></name><name><surname>Iqbal</surname><given-names>Z</given-names></name><name><surname>Hunt</surname><given-names>M</given-names></name><name><surname>Smith</surname><given-names>EG</given-names></name><name><surname>Rathod</surname><given-names>P</given-names></name><name><surname>Jarrett</surname><given-names>L</given-names></name><name><surname>Matias</surname><given-names>D</given-names></name><name><surname>Cirillo</surname><given-names>DM</given-names></name><name><surname>Borroni</surname><given-names>E</given-names></name><name><surname>Battaglia</surname><given-names>S</given-names></name><name><surname>Chiacchiaretta</surname><given-names>M</given-names></name><name><surname>De Filippo</surname><given-names>M</given-names></name><name><surname>Cabibbe</surname><given-names>A</given-names></name><name><surname>Tahseen</surname><given-names>S</given-names></name><name><surname>Mistry</surname><given-names>N</given-names></name><name><surname>Nilgiriwala</surname><given-names>K</given-names></name><name><surname>Chitalia</surname><given-names>V</given-names></name><name><surname>Ganesan</surname><given-names>N</given-names></name><name><surname>Papewar</surname><given-names>A</given-names></name><name><surname>Rodrigues</surname><given-names>C</given-names></name><name><surname>Kambli</surname><given-names>P</given-names></name><name><surname>Surve</surname><given-names>U</given-names></name><name><surname>Khot</surname><given-names>R</given-names></name><name><surname>Niemann</surname><given-names>S</given-names></name><name><surname>Kohl</surname><given-names>T</given-names></name><name><surname>Merker</surname><given-names>M</given-names></name><name><surname>Hoffmann</surname><given-names>H</given-names></name><name><surname>Lehmann</surname><given-names>S</given-names></name><name><surname>Plesnik</surname><given-names>S</given-names></name><name><surname>Ismail</surname><given-names>N</given-names></name><name><surname>Omar</surname><given-names>SV</given-names></name><name><surname>Joseph</surname><given-names>L</given-names></name><name><surname>Marubini</surname><given-names>E</given-names></name><name><surname>Thwaites</surname><given-names>G</given-names></name><name><surname>Thuy Thuong</surname><given-names>TN</given-names></name><name><surname>Ngoc</surname><given-names>NH</given-names></name><name><surname>Srinivasan</surname><given-names>V</given-names></name><name><surname>Moore</surname><given-names>D</given-names></name><name><surname>Coronel</surname><given-names>J</given-names></name><name><surname>Solano</surname><given-names>W</given-names></name><name><surname>He</surname><given-names>G</given-names></name><name><surname>Zhu</surname><given-names>B</given-names></name><name><surname>Zhou</surname><given-names>Y</given-names></name><name><surname>Ma</surname><given-names>A</given-names></name><name><surname>Yu</surname><given-names>P</given-names></name><name><surname>Schito</surname><given-names>M</given-names></name><name><surname>Claxton</surname><given-names>P</given-names></name><name><surname>Laurenson</surname><given-names>I</given-names></name></person-group><year iso-8601-date="2018">2018</year><article-title>Antimicrobial Agents and Chemotherapy</article-title><source>ASM Journals Logo</source><volume>62</volume><elocation-id>e00344–18</elocation-id></element-citation></ref><ref id="bib18"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Schön</surname><given-names>T</given-names></name><name><surname>Matuschek</surname><given-names>E</given-names></name><name><surname>Mohamed</surname><given-names>S</given-names></name><name><surname>Utukuri</surname><given-names>M</given-names></name><name><surname>Heysell</surname><given-names>S</given-names></name><name><surname>Alffenaar</surname><given-names>J-W</given-names></name><name><surname>Shin</surname><given-names>S</given-names></name><name><surname>Martinez</surname><given-names>E</given-names></name><name><surname>Sintchenko</surname><given-names>V</given-names></name><name><surname>Maurer</surname><given-names>FP</given-names></name><name><surname>Keller</surname><given-names>PM</given-names></name><name><surname>Kahlmeter</surname><given-names>G</given-names></name><name><surname>Köser</surname><given-names>CU</given-names></name></person-group><year iso-8601-date="2019">2019</year><article-title>Standards for MIC testing that apply to the majority of bacterial pathogens should also be enforced for Mycobacterium tuberculosis complex</article-title><source>Clinical Microbiology and Infection</source><volume>25</volume><fpage>403</fpage><lpage>405</lpage><pub-id pub-id-type="doi">10.1016/j.cmi.2019.01.019</pub-id><pub-id pub-id-type="pmid">30771527</pub-id></element-citation></ref><ref id="bib19"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Schön</surname><given-names>T</given-names></name><name><surname>Werngren</surname><given-names>J</given-names></name><name><surname>Machado</surname><given-names>D</given-names></name><name><surname>Borroni</surname><given-names>E</given-names></name><name><surname>Wijkander</surname><given-names>M</given-names></name><name><surname>Lina</surname><given-names>G</given-names></name><name><surname>Mouton</surname><given-names>J</given-names></name><name><surname>Matuschek</surname><given-names>E</given-names></name><name><surname>Kahlmeter</surname><given-names>G</given-names></name><name><surname>Giske</surname><given-names>C</given-names></name><name><surname>Santin</surname><given-names>M</given-names></name><name><surname>Cirillo</surname><given-names>DM</given-names></name><name><surname>Viveiros</surname><given-names>M</given-names></name><name><surname>Cambau</surname><given-names>E</given-names></name></person-group><year iso-8601-date="2020">2020</year><article-title>Antimicrobial susceptibility testing of Mycobacterium tuberculosis complex isolates – the EUCAST broth microdilution reference method for MIC determination</article-title><source>Clinical Microbiology and Infection</source><volume>27</volume><fpage>10</fpage><lpage>13</lpage><pub-id pub-id-type="doi">10.1016/j.cmi.2020.10.019</pub-id><pub-id pub-id-type="pmid">33198949</pub-id></element-citation></ref><ref id="bib20"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Spiers</surname><given-names>H</given-names></name><name><surname>Swanson</surname><given-names>A</given-names></name><name><surname>Fortson</surname><given-names>L</given-names></name><name><surname>Simmons</surname><given-names>BD</given-names></name><name><surname>Trouille</surname><given-names>L</given-names></name><name><surname>Blickhan</surname><given-names>S</given-names></name><name><surname>Lintott</surname><given-names>C</given-names></name></person-group><year iso-8601-date="2019">2019</year><article-title>Everyone counts? Design considerations in online citizen science</article-title><source>Journal of Science Communication</source><volume>18</volume><elocation-id>A04</elocation-id><pub-id pub-id-type="doi">10.22323/2.18010204</pub-id></element-citation></ref><ref id="bib21"><element-citation publication-type="preprint"><person-group person-group-type="author"><name><surname>Spiers</surname><given-names>H</given-names></name><name><surname>Songhurst</surname><given-names>H</given-names></name><name><surname>Nightingale</surname><given-names>L</given-names></name><name><surname>de Folter</surname><given-names>J</given-names></name><name><surname>Hutchings</surname><given-names>R</given-names></name><name><surname>Peddie</surname><given-names>CJ</given-names></name><name><surname>Weston</surname><given-names>A</given-names></name><name><surname>Strange</surname><given-names>A</given-names></name><name><surname>Hindmarsh</surname><given-names>S</given-names></name><name><surname>Lintott</surname><given-names>C</given-names></name><name><surname>Collinson</surname><given-names>LM</given-names></name><name><surname>Jones</surname><given-names>ML</given-names></name></person-group><year iso-8601-date="2020">2020</year><article-title>Citizen Science, Cells and CNNs – Deep Learning for Automatic Segmentation of the Nuclear Envelope in Electron Microscopy Data, Trained with Volunteer Segmentations</article-title><source>bioRxiv</source><pub-id pub-id-type="doi">10.1101/2020.07.28.223024</pub-id></element-citation></ref><ref id="bib22"><element-citation publication-type="preprint"><person-group person-group-type="author"><collab>The CRyPTIC Consortium</collab></person-group><year iso-8601-date="2021">2021</year><article-title>A Crowd of BashTheBug Volunteers Reproducibly and Accurately Measure the Minimum Inhibitory Concentrations of 13 Antitubercular Drugs from Photographs of 96-Well Broth Microdilution Plates</article-title><source>bioRxiv</source><pub-id pub-id-type="doi">10.1101/20210914460353</pub-id></element-citation></ref><ref id="bib23"><element-citation publication-type="preprint"><person-group person-group-type="author"><collab>The CRyPTIC Consortium</collab></person-group><year iso-8601-date="2022">2022</year><article-title>BashTheBug: A Crowd of Volunteers Reproducibly and Accurately Measure the Minimum Inhibitory Concentrations of 13 Antitubercular Drugs from Photographs of 96-Well Broth Microdilution Plates</article-title><source>bioRxiv</source><pub-id pub-id-type="doi">10.1101/20210914460274</pub-id></element-citation></ref><ref id="bib24"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Trouille</surname><given-names>L</given-names></name><name><surname>Lintott</surname><given-names>CJ</given-names></name><name><surname>Fortson</surname><given-names>LF</given-names></name></person-group><year iso-8601-date="2019">2019</year><article-title>Citizen science frontiers: Efficiency, engagement, and serendipitous discovery with human-machine systems</article-title><source>PNAS</source><volume>116</volume><fpage>1902</fpage><lpage>1909</lpage><pub-id pub-id-type="doi">10.1073/pnas.1807190116</pub-id><pub-id pub-id-type="pmid">30718393</pub-id></element-citation></ref><ref id="bib25"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Walker</surname><given-names>TM</given-names></name><name><surname>Cruz</surname><given-names>ALG</given-names></name><name><surname>Peto</surname><given-names>TE</given-names></name><name><surname>Smith</surname><given-names>EG</given-names></name><name><surname>Esmail</surname><given-names>H</given-names></name><name><surname>Crook</surname><given-names>DW</given-names></name></person-group><year iso-8601-date="2017">2017</year><article-title>Tuberculosis is changing</article-title><source>The Lancet. Infectious Diseases</source><volume>17</volume><fpage>359</fpage><lpage>361</lpage><pub-id pub-id-type="doi">10.1016/S1473-3099(17)30123-8</pub-id><pub-id pub-id-type="pmid">28298254</pub-id></element-citation></ref><ref id="bib26"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Walker</surname><given-names>TM</given-names></name><name><surname>Miotto</surname><given-names>P</given-names></name><name><surname>Köser</surname><given-names>CU</given-names></name><name><surname>Fowler</surname><given-names>PW</given-names></name><name><surname>Knaggs</surname><given-names>J</given-names></name><name><surname>Iqbal</surname><given-names>Z</given-names></name><name><surname>Hunt</surname><given-names>M</given-names></name><name><surname>Chindelevitch</surname><given-names>L</given-names></name><name><surname>Farhat</surname><given-names>M</given-names></name><name><surname>Cirillo</surname><given-names>DM</given-names></name><name><surname>Comas</surname><given-names>I</given-names></name><name><surname>Posey</surname><given-names>J</given-names></name><name><surname>Omar</surname><given-names>SV</given-names></name><name><surname>Peto</surname><given-names>TE</given-names></name><name><surname>Suresh</surname><given-names>A</given-names></name><name><surname>Uplekar</surname><given-names>S</given-names></name><name><surname>Laurent</surname><given-names>S</given-names></name><name><surname>Colman</surname><given-names>RE</given-names></name><name><surname>Nathanson</surname><given-names>C-M</given-names></name><name><surname>Zignol</surname><given-names>M</given-names></name><name><surname>Walker</surname><given-names>AS</given-names></name><name><surname>Crook</surname><given-names>DW</given-names></name><name><surname>Ismail</surname><given-names>N</given-names></name><name><surname>Rodwell</surname><given-names>TC</given-names></name><collab>CRyPTIC Consortium</collab><collab>Seq&Treat Consortium</collab></person-group><year iso-8601-date="2022">2022</year><article-title>The 2021 WHO catalogue of <italic>Mycobacterium tuberculosis</italic> complex mutations associated with drug resistance: A genotypic analysis</article-title><source>The Lancet. Microbe</source><volume>3</volume><fpage>e265</fpage><lpage>e273</lpage><pub-id pub-id-type="doi">10.1016/S2666-5247(21)00301-3</pub-id><pub-id pub-id-type="pmid">35373160</pub-id></element-citation></ref><ref id="bib27"><element-citation publication-type="report"><person-group person-group-type="author"><collab>World Health Organization</collab></person-group><year iso-8601-date="2020">2020</year><source>Global Tuberculosis Report, Technical report</source><publisher-name>WHO</publisher-name></element-citation></ref><ref id="bib28"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Zhu</surname><given-names>T</given-names></name><name><surname>Johnson</surname><given-names>AEW</given-names></name><name><surname>Behar</surname><given-names>J</given-names></name><name><surname>Clifford</surname><given-names>GD</given-names></name></person-group><year iso-8601-date="2014">2014</year><article-title>Crowd-sourced annotation of ecg signals using contextual information</article-title><source>Annals of Biomedical Engineering</source><volume>42</volume><fpage>871</fpage><lpage>884</lpage><pub-id pub-id-type="doi">10.1007/s10439-013-0964-6</pub-id><pub-id pub-id-type="pmid">24368593</pub-id></element-citation></ref></ref-list></back><sub-article article-type="editor-report" id="sa0"><front-stub><article-id pub-id-type="doi">10.7554/eLife.75046.sa0</article-id><title-group><article-title>Editor's evaluation</article-title></title-group><contrib-group><contrib contrib-type="author"><name><surname>Kana</surname><given-names>Bavesh D</given-names></name><role specific-use="editor">Reviewing Editor</role><aff><institution-wrap><institution-id institution-id-type="ror">https://ror.org/03rp50x72</institution-id><institution>University of the Witwatersrand</institution></institution-wrap><country>South Africa</country></aff></contrib></contrib-group><related-object id="sa0ro1" link-type="continued-by" object-id="10.1101/2021.07.20.453060" object-id-type="id" xlink:href="https://sciety.org/articles/activity/10.1101/2021.07.20.453060"/></front-stub><body><p>The authors evaluate a novel crowd–sourcing method to interpret minimum inhibitory concentrations of <italic>Mycobacterium tuberculosis</italic>, the causative agent of tuberculosis. To provide valuable test results without the need for available expert mycobacteriologists, the authors demonstrate that when presented appropriately, 11–17 interpretations by lay interpreters can provide reproducible results for most tuberculosis drugs. This analysis demonstrates that among those samples that can be reliably interpreted by automated detection software, lay interpretation provides a potential alternative means to provide a timely confirmatory read. The work will be of interest to bacteriologists and those with an interest in antimicrobial resistance.</p></body></sub-article><sub-article article-type="decision-letter" id="sa1"><front-stub><article-id pub-id-type="doi">10.7554/eLife.75046.sa1</article-id><title-group><article-title>Decision letter</article-title></title-group><contrib-group content-type="section"><contrib contrib-type="editor"><name><surname>Kana</surname><given-names>Bavesh D</given-names></name><role>Reviewing Editor</role><aff><institution-wrap><institution-id institution-id-type="ror">https://ror.org/03rp50x72</institution-id><institution>University of the Witwatersrand</institution></institution-wrap><country>South Africa</country></aff></contrib></contrib-group></front-stub><body><boxed-text id="sa2-box1"><p>Our editorial process produces two outputs: i) <ext-link ext-link-type="uri" xlink:href="https://sciety.org/articles/activity/10.1101/2021.07.20.453060">public reviews</ext-link> designed to be posted alongside <ext-link ext-link-type="uri" xlink:href="https://www.biorxiv.org/content/10.1101/2021.07.20.453060v3">the preprint</ext-link> for the benefit of readers; ii) feedback on the manuscript for the authors, including requests for revisions, shown below. We also include an acceptance summary that explains what the editors found interesting or important about the work.</p></boxed-text><p><bold>Decision letter after peer review:</bold></p><p>Thank you for submitting your article "BashTheBug: a crowd of volunteers reproducibly and accurately measure the minimum inhibitory concentrations of 13 antitubercular drugs from photographs of 96–well broth microdilution plates" for consideration by <italic>eLife</italic>. Your article has been reviewed by 3 peer reviewers, and the evaluation has been overseen by Bavesh Kana as the Senior and Reviewing Editor. The reviewers have opted to remain anonymous.</p><p>The reviewers have discussed their reviews with one another, and the Reviewing Editor has drafted this to help you prepare a revised submission.</p><p>Essential revisions:</p><p>Data presentation/interpretation/clarity:</p><p>1. While the authors explained how they try to engage people to "play" this serious game and describe the number of classifications, there is no real discussion about the number of users playing every day. Reaching a minimum number of regular players is essential for the sustainability of such project.</p><p>2. In the discussion, the authors mentioned that this approach may help training laboratory scientists unfortunately this claim was not really explored in this manuscript. It may have been interesting to analyze, for the most engaged volunteers, the improvement in term of accuracy of a user after 10, 100 or 1000 classifications. This may be also interesting to reweight the accuracy results in function of the users experience to see if it can improve the classification scores. It would have been also of interest to know if the way of presenting the data or playing the game may help experts (i.e. laboratory scientists) to improve their skills to quickly assess MIC using the methodology design to assess the citizens (such as time spent on a classification presented in Figure S5).</p><p>3. 13 drugs were tested on 19 different strains of Mtb. It would have been of broad interest to see then how to reconstruct each plate from the different classifications and briefly present the practical outputs of these classifications: i.e. the resistance of each strain to the different antibiotics. Furthermore, except H37rV, the other strains are not mentioned; only a vial code is presented in Table S1.</p><p>4. Explain why you decided on 17 classifications in particular? You seem to make several arguments for 9 or even 3 classifications, but the reason for 17 in particular is not clear. Why do some wells have more than 17 classifications: was this accident or design? If design, was it because of a lack of consensus about them, or some other reason?</p><p>Editorial:</p><p>1. In the abstract (lines 3–4), the authors refer to MIC testing as a conventional method, which is imprecise, as the majority of BSL3 labs that perform susceptibility testing for Mtb only assess a single critical concentration per drug. This is also a challenge for the Introduction, paragraph 3.</p><p>2. In the introduction p. 2 the authors mentioned: "(ii) to provide a large dataset of classifications to train machine–learning models". While, one can see how the datasets can be used for this purpose, there is no data in the current manuscript corroborating this claim. This sentence at this position in the introduction may need to be reformulate otherwise it can be seen as misleading for the reader looking for data about ML model training.</p><p>3. First paragraph: two very minor linguistic points, (1) Do you mean 1.4 million deaths worldwide in 2019, rather than restricted to a particular location, and this might provide additional context? (2) in the sentence "Ordinarily this is more than any other single pathogen, however SARS–CoV–2 killed more people than TB in 2020 and is likely to do so again in 2021", the flow of the paper would be clearer if the word "however" was replaced with "although" – it sounds as if the subject is about to change to the SARS–CoV–2 virus as it is.</p><p>4. Introduction, paragraph 1, line 3–4 – not all bacterial pathogens demonstrate increasing resistance, perhaps the term "most" would be more accurate</p><p>5. Are the datasets (Expert and Expert+AMyGDA) are available for reuse especially to see if other programs can compare with the current results?</p><p>6. p. 5 "however some images attracted additional classifications with 421 images having > 34 classifications". Can the authors comments why? Based on beginning of the sentence, one gets the sense that the impression that the images were retired after 17 classifications?</p><p>7. p. 8 Figure 1–A shows that the classifications were condensed in two periods and a long gap with no classifications from October 2017 to April 2019. Can the authors explain why? In the panel C, there is a bump at the duration equal to 25 s. Is there something happening in the program at that time?</p><p>8. p. 12 Figure 3, why is panel E is positioned in a blank square on panel D?</p><p>9. Methods, Image Dataset, paragraph 3, line 1: "asking a volunteers" should be changed to "asking volunteers" or "asking a volunteer"</p><p>10. Results, Time Spent, line 2: would rephrase, the "caused by e.g." section for improved readability.</p><p>11. Results, Expert Measurement, paragraph 3, line 2: the phrase "one candidate" is confusing, suggest "one possibility" or other phrasing.</p><p>12. Results, Variation by Drug: an additional consideration could be the role of drug position on the plate, as some drugs (such as delamanid) are known to degrade faster, and the central drugs of the plate appeared to be those with lower agreement. This could potentially be missed due to interpretation of higher than normal susceptibility to delamanid (as could be artificially elevated if the lyophilized drug on the plate were used >6 months from manufacture). That said, this is very hard to differentiate given that the entire plate, as discussed, is tested as a single unit, and the dataset represented limited resistance to clofazimine, bedaquiline, delamanid, and linezolid. Can you comment on this?</p><p>13. Are the authors willing to open the BashtheBug platform so that other labs – especially in developing countries – can provide datasets?</p><p>14. The mention that CRyPTIC is seeking to expand knowledge of genetic variants of TB and the reasons for this are very exciting. However, this sentiment is not echoed again in the manuscript. Some elaboration of this, in the context of your findings, might greatly strengthen your discussion, if you feel it is appropriate. For example, based on what the volunteers can do, what work do you plan to do based on it, have you learned anything new about the different genomes, and do you have any recommendations for other projects sequencing microbial genomes?</p><p>15. It would have been extremely interesting, if at all possible, to see what would have happened if several experts had classified the same plate – what might the errors have been then? You note that experts are inconsistent, but it might be interesting to calculate how much so, compared to volunteers. Can you comment on this?</p><p>16. Please explain why the intervals of 7, 10, 14 and 21 days of incubation were chosen for the analysis of the cultures? (The paper does not mention that TB cultures grow unusually slowly, and that of course this is an additional reason why treatment is difficult). Readers working in the field will doubtless be aware of this fact, but those in citizen science or other disciplines may not be.</p><p>17. Can you describe where the 20 000 cultures came from?</p><p>18. You describe, "Thirty one vials containing 19 external quality assessment (EQA) <italic>M. tuberculosis</italic> strains, including the reference strain H37Rv ATCC 272948", which "were sent to seven participating laboratories". How were these selected? Please add a summary of how this process took place. In the text, this paragraph is confusing because seven separate institutions were mentioned but only two members of staff – did these staff travel between institutions?</p><p>19. The paper would be clearer if there had been a separate section specifically on comparing Volunteer measurements with Expert and/or with Expert+AMyGDA.</p><p>20. Figure 1 (A, B and C) is too small.</p><p>21. It would be very useful to explain a little more fully exactly why "This is slightly less than the 95% threshold set by ISO for aerobic bacteria and therefore the volunteers do not need this criterion". The sentence implies the opposite, given that the volunteers do not meet the 95% threshold, this is exactly why they would need this criterion. Some clarification here might strengthen the main finding, that a crowd of volunteers can perform the required task.</p><p>22. It may possibly be worth quoting or referencing some literature on "the wisdom of crowds" or some such topic to explain more fully why you chose citizen science, and a large number of untrained eyes rather than a small number of trained ones, to analyse your dataset.</p><p>23. The authors mention several biases, e.g. the different error sources in the "expert" versus the "AMyGDA" analyses, and in the tendency of volunteers to pick high MICs while the AMyGDA picks low ones, but overall throughout the paper it is less clear how these were actually corrected for or otherwise how these bias were addressed.</p><p><italic>Reviewer #2 (Recommendations for the authors):</italic></p><p>Thank you for the opportunity to review your paper. I want to stress before I make my comments that I do lack expertise in the field of microbiology, and some of what I suggest might very justifiably be discounted on this basis. I cannot provide comments on your reagents or equipment, for example, but perhaps I can make some useful suggestions on paper clarity and the citizen science aspects.</p><p>Introduction:</p><p>First paragraph: two very minor linguistic points, (1) I presume you mean 1.4 million deaths worldwide in 2019, rather than restricted to a particular location, and this might provide additional context, (2) in the sentence "Ordinarily this is more than any other single pathogen, however SARS–CoV–2 killed more people than TB in 2020 and is likely to do so again in 2021", I feel the flow of the paper would be clearer if the word "however" was replaced with "although" – it sounds as if the subject is about to change to the SARS–CoV–2 virus as it is.</p><p>The mention that CRyPTIC is seeking to expand knowledge of genetic variants of TB and the reasons for this are very exciting. However, you don't mention them again. Some elaboration of this, in the context of your findings, might greatly strengthen your discussion, if you feel it is appropriate. For example, based on what the volunteers can do, what work do you plan to do based on it, have you learned anything new about the different genomes, and do you have any recommendations for other projects sequencing microbial genomes?</p><p>Methods:</p><p>I particularly appreciated the point about, having found that classifying 96 wells at a time was too arduous a task, the authors decided to watch what an expert did. It sounds as if the authors were willing to try out several methods before settling on one which was appropriate for volunteers. (Similarly, your sections describing "Engagement", for example, and from the looks of your social media, sound like you established a lively community, which is a great thing in citizen science.) This paragraph is not a recommendation to change anything – except to say that I hope you will do further citizen science projects in the future!</p><p>It would have been extremely interesting, if at all possible, to see what would have happened if several experts had classified the same plate – what might the errors have been then? You note that experts are inconsistent, but it might be interesting to calculate how much so, compared to volunteers. I do not suggest you redo any experiments; perhaps a short speculation or any findings you do have, even anecdotal, might suffice to provide insights here.</p><p>Please could an explanation be added of why you decided on 17 classifications in particular? You seem to make several arguments for 9 or even 3 classifications, and this comes into various discussions several times, but the reason for 17 in particular never became quite clear to me. I would also have been very interested to know why some wells had more than 17 classifications: was this accident or design? If design, was it because of a lack of consensus about them, or some other reason?</p><p>Please could an explanation also be added of why the intervals of 7, 10, 14 and 21 days of incubation were chosen for the analysis of the cultures? (The paper does not mention that TB cultures grow unusually slowly, and that of course this is an additional reason why treatment is difficult. Readers working in the field will doubtless be aware of this fact, but those in citizen science or other disciplines may not be, and I feel this paper will be valuable to those in other disciplines.) I noticed there were a great many references to classifications for 14 days, but very little seemed to be said about 7, 10 and 21 in comparison. I did not feel that Figure S12 really explained how the images themselves might change over time due to bacterial growth, or why 7, 10 and 21 days were paid comparatively little attention.</p><p>I would have felt reassured to know where CRyPTIC's 20,000 cultures came from. I presume they were taken from patients from a range of countries. Was patient consent required, or obtained, for their use in this project? I presume that procedures were followed, but I would recommend even a very brief clarification of this point.</p><p>Image Dataset:</p><p>I did not find it not clear whether the "Thirty one vials containing 19 external quality assessment (EQA) <italic>M. tuberculosis</italic> strains, including the reference strain H37Rv ATCC 272948", which "were sent to seven participating laboratories", were part of the 20,000–strong dataset or how exactly they were selected from this larger sample. The phrase which immediately follows this, "sent to seven participating laboratories as described previously", in fact refers to another paper, but I felt that the phrase "as described previously" hints that you describe it in this paper. I recommend that you add a summary of how this process took place, even a very short sentence. Additionally, I found this paragraph confusing because seven separate institutions were mentioned but only two members of staff – did these staff travel between institutions?</p><p>There is a clear section on "Expert Measurements", but I would have found the paper clearer if there had been a separate section specifically on comparing Volunteer measurements with Expert and/or with Expert+AMyGDA. (The section "Variability in classifications" certainly goes into this, but it felt to me that it brought together several issues at once, which made it less easy to understand.)</p><p>"The AMyGDA software was therefore modified to composite such drug images" is a sentence I do not quite understand, even when I look at Figure 1B. Does it mean that several images were stacked on top of each other? Or did it simply mean – as I generally understand – that the volunteers are not shown the entire image in one go, but rather they are shown a small section of it with a line of wells for each particular type of antibiotic plus the control?</p><p>I found Figure 1 (A, B and C) much too small to really be able to see what was happening clearly. On print, it would be very difficult to see! The later figures illustrated the methods and the Zooniverse interface far better. Figure 1 is very valuable and I recommend the authors consider expanding it, even if it takes up more space.</p><p>"How to compare?":</p><p>I had difficulty understanding the terms "exact agreement" and "essential agreement", which may simply be due to my own limited knowledge. However, I found myself doing a literature search for the terms and found nothing – which is understandable, given that you have created new key metrics. I do not quite understand the need for new key metrics – perhaps the authors might consider stating why established metrics were inadequate in this case. I found it difficult to find the reference provided in the same paragraph regarding the creation of new AST methods, as there were a great many with much the same name, the first one of which was a withdrawn paper (written in 2007 but revised in 2021), so a DOI in the reference list might be helpful here if you feel it is worthwhile.</p><p>Discussion:</p><p>It would be very useful to explain a little more fully exactly why "This is slightly less than the 95% threshold set by ISO for aerobic bacteria and therefore the volunteers do not need this criterion". The sentence implies the opposite, given that the volunteers do not meet the 95% threshold, this is exactly why they would need this criterion. Some clarification here might strengthen the main finding, that a crowd of volunteers can perform the required task.</p><p>I was very interested by the point that "… one could build a hybrid approach where e.g. small crowds of experts could examine plates used in a clinical microbiology service – these could be particularly difficult drug images or could be a random sample for quality assurance purposes." I recommend that random samples are always included in citizen science projects even when you want human volunteers to focus on difficult images (so that artificial intelligence can do the easier ones), because training volunteers only on difficult images means that they have no clear idea what a representative sample looks like and this could create a bias or other problems. There was an interesting case in the Zooniverse project Snapshot Serengeti, in which volunteers were asked to classify animals in photographs taken by camera traps. Some photographs only showed grass and no animals, and these images were removed because of the ethic of not wasting volunteer time. However, surprisingly, this actually decreased volunteer engagement, because essentially if every image is special then no image is special. (The Snapshot Serengeti team produced a poster about this, but to the best of my knowledge it was not written up in an academic paper – however, the Zooniverse team will probably be able to confirm that these events took place and provide you with a reference.) It is important that a human volunteer knows what a "weird" image is and how it is different from a standard one.</p><p>General points:</p><p>It may possibly be worth quoting or referencing some literature on "the wisdom of crowds" or some such topic to explain more fully why you chose citizen science, and a large number of untrained eyes rather than a small number of trained ones, to analyse your dataset.</p><p>The authors mention several biases, e.g. the different error sources in the "expert" versus the "AMyGDA" analyses, and in the tendency of volunteers to pick high MICs while the AMyGDA picks low ones, but overall throughout the paper I sometimes felt less clear on how these were actually corrected for or otherwise how these bias were addressed.</p><p>Overall, these are very minor criticisms of what to me was a very interesting paper. I wish you all the best with its publication, and for your future work building on these results.</p></body></sub-article><sub-article article-type="reply" id="sa2"><front-stub><article-id pub-id-type="doi">10.7554/eLife.75046.sa2</article-id><title-group><article-title>Author response</article-title></title-group></front-stub><body><disp-quote content-type="editor-comment"><p>Essential revisions:</p><p>Data presentation/interpretation/clarity:</p><p>1. While the authors explained how they try to engage people to "play" this serious game and describe the number of classifications, there is no real discussion about the number of users playing every day. Reaching a minimum number of regular players is essential for the sustainability of such project.</p></disp-quote><p>This is a good point. We have added a small panel to Figure 2 showing the distribution of number of active users per day and some text to the Volunteers subsection in the Results.</p><disp-quote content-type="editor-comment"><p>2. In the discussion, the authors mentioned that this approach may help training laboratory scientists unfortunately this claim was not really explored in this manuscript. It may have been interesting to analyze, for the most engaged volunteers, the improvement in term of accuracy of a user after 10, 100 or 1000 classifications. This may be also interesting to reweight the accuracy results in function of the users experience to see if it can improve the classification scores. It would have been also of interest to know if the way of presenting the data or playing the game may help experts (i.e. laboratory scientists) to improve their skills to quickly assess MIC using the methodology design to assess the citizens (such as time spent on a classification presented in Figure S5).</p></disp-quote><p>We have calculated the average bias for each volunteer (defined as the mean of all the differences between their dilution and the dilution of the Expert+AMyGDA dataset) and then plotted this against the total number of classifications each volunteer has made (Figure S17). This suggests that as volunteers get more experienced their bias reduces but does not disappear and we have added a sentence to the Discussion making this point and referring to the new figure. We did consider reweighting the scores but to have done this we would have needed to get all volunteers to classify a small set of known images to establish their weight which we would then prospectively apply to their classifications.</p><disp-quote content-type="editor-comment"><p>3. 13 drugs were tested on 19 different strains of Mtb. It would have been of broad interest to see then how to reconstruct each plate from the different classifications and briefly present the practical outputs of these classifications: i.e. the resistance of each strain to the different antibiotics. Furthermore, except H37rV, the other strains are not mentioned; only a vial code is presented in Table S1.</p></disp-quote><p>This is related to §2.16, §2.18. The resistance profile of the 19 EQA strains is described in detail in a previous study (https://doi.org/10.1128/AAC.00344-18) which we did not flag well enough in the manuscript. We’ve added a sentence pointing out that the EQA strains had different resistance profiles and cited the above reference.</p><disp-quote content-type="editor-comment"><p>4. Explain why you decided on 17 classifications in particular? You seem to make several arguments for 9 or even 3 classifications, but the reason for 17 in particular is not clear. Why do some wells have more than 17 classifications: was this accident or design? If design, was it because of a lack of consensus about them, or some other reason?</p></disp-quote><p>We wanted each image to be classified enough times so we could analyse the effect of number of classifications on reproducibility and accuracy but had to balance this against the ethics of asking our volunteers to do classifications that were essentially spurious i.e. their work was not changing our conclusions. An odd number was chosen to make finding the median easier.</p><disp-quote content-type="editor-comment"><p>Editorial:</p><p>1. In the abstract (lines 3–4), the authors refer to MIC testing as a conventional method, which is imprecise, as the majority of BSL3 labs that perform susceptibility testing for Mtb only assess a single critical concentration per drug. This is also a challenge for the Introduction, paragraph 3.</p></disp-quote><p>The reviewers are correct, these are imprecise statements. We have reworded parts of the abstract and the introductions to correct this.</p><disp-quote content-type="editor-comment"><p>2. In the introduction p. 2 the authors mentioned: "(ii) to provide a large dataset of classifications to train machine–learning models". While, one can see how the datasets can be used for this purpose, there is no data in the current manuscript corroborating this claim. This sentence at this position in the introduction may need to be reformulate otherwise it can be seen as misleading for the reader looking for data about ML model training.</p></disp-quote><p>We agree this is potentially confusing and any discussion about using the dataset for machine-learning is best left to the Discussion so have removed this the mention of machine-learning from the Introduction.</p><disp-quote content-type="editor-comment"><p>3. First paragraph: two very minor linguistic points, (1) Do you mean 1.4 million deaths worldwide in 2019, rather than restricted to a particular location, and this might provide additional context? (2) in the sentence "Ordinarily this is more than any other single pathogen, however SARS–CoV–2 killed more people than TB in 2020 and is likely to do so again in 2021", the flow of the paper would be clearer if the word "however" was replaced with "although" – it sounds as if the subject is about to change to the SARS–CoV–2 virus as it is.</p></disp-quote><p>Yes, 1.4 million people were estimated to have died worldwide from TB in 2019. Agree with both points and have changed the text.</p><disp-quote content-type="editor-comment"><p>4. Introduction, paragraph 1, line 3–4 – not all bacterial pathogens demonstrate increasing resistance, perhaps the term "most" would be more accurate</p></disp-quote><p>Agreed, changed.</p><disp-quote content-type="editor-comment"><p>5. Are the datasets (Expert and Expert+AMyGDA) are available for reuse especially to see if other programs can compare with the current results?</p></disp-quote><p>Yes, the GitHub repository at https://github.com/fowler-lab/bashthebugconsensus-dataset contains the data tables (as CSVs) in the tables/ folder and there is a data schema provided. The PHENOTYPES table contains one row per drug per sample per reading day and has columns for each of the different reading methods (this is explained in the README.md file). For example, the Expert reading (given as a dilution) is recorded in the VZ column and the AMyGDA reading is recorded in the IM column. These are given as dilutions i.e. the MIC is the number of the well counting up from the lowest concentration starting at one. A negative number indicates that either the control wells on the plate failed or that this drug could not be read due to e.g. skip wells or contamination.</p><disp-quote content-type="editor-comment"><p>6. p. 5 "however some images attracted additional classifications with 421 images having > 34 classifications". Can the authors comments why? Based on beginning of the sentence, one gets the sense that the impression that the images were retired after 17 classifications?</p></disp-quote><p>We think this is because when an image reaches the retirement limit it is not removed from the pool of images shown to the volunteers but instead a “Finished!” sticker is shown on the top left part of the image. Hence it is possible for images to be classified more times than the retirement limit and this is likely to happen when all the current uploaded images have been retired. We have spoken to the Zooniverse team about this, and they are aware, but it is unclear to us if, in this situation, some images are more likely than others to be served to volunteers thereby exacerbating the problem, or whether it is purely random.</p><p>We have added a sentence explaining this to the text.</p><disp-quote content-type="editor-comment"><p>7. p. 8 Figure 1–A shows that the classifications were condensed in two periods and a long gap with no classifications from October 2017 to April 2019. Can the authors explain why? In the panel C, there is a bump at the duration equal to 25 s. Is there something happening in the program at that time?</p></disp-quote><p>During the first period the retirement limit was set to eleven (which is also the Zooniverse default value), however during early analysis of the data it became apparent that we would need additional classifications so we could analyse in more detail the effect of adding additional classifications, hence the second period. In the period in between the volunteers were working on images taken from real plates collected by the project.</p><p>We don’t know why there is a bump at 25 seconds! We suspect, but do not know for sure, that it is a rounding artefact. Since this duration is much greater than the modal duration of</p><p>3.5 seconds, we have not commented further on it in the manuscript.</p><disp-quote content-type="editor-comment"><p>8. p. 12 Figure 3, why is panel E is positioned in a blank square on panel D?</p></disp-quote><p>In Figure 3D each cell in the heatmap is annotated with the percentage of images if it is greater than 0.1%. Hence Figure 3E is pointing at a cell which has less than 0.1% of images and so appears blank but actually contains a number of classifications. We’ve made this clearer in the legend of Figure 3.</p><disp-quote content-type="editor-comment"><p>9. Methods, Image Dataset, paragraph 3, line 1: "asking a volunteers" should be changed to "asking volunteers" or "asking a volunteer"</p></disp-quote><p>Changed, thanks.</p><disp-quote content-type="editor-comment"><p>10. Results, Time Spent, line 2: would rephrase, the "caused by e.g." section for improved readability.</p></disp-quote><p>Thanks, have rephrased.</p><disp-quote content-type="editor-comment"><p>11. Results, Expert Measurement, paragraph 3, line 2: the phrase "one candidate" is confusing, suggest "one possibility" or other phrasing.</p></disp-quote><p>Have rephrased as suggested.</p><disp-quote content-type="editor-comment"><p>12. Results, Variation by Drug: an additional consideration could be the role of drug position on the plate, as some drugs (such as delamanid) are known to degrade faster, and the central drugs of the plate appeared to be those with lower agreement. This could potentially be missed due to interpretation of higher than normal susceptibility to delamanid (as could be artificially elevated if the lyophilized drug on the plate were used >6 months from manufacture). That said, this is very hard to differentiate given that the entire plate, as discussed, is tested as a single unit, and the dataset represented limited resistance to clofazimine, bedaquiline, delamanid, and linezolid. Can you comment on this?</p></disp-quote><p>This is a good question and was explicitly tackled by an earlier paper that sought to validate the UKMYC5 plate for use in research antimicrobial susceptibility testing (doi: 10.1128/AAC.00344-18) – we have therefore added a direct comparison to the results in that study and cited it.</p><disp-quote content-type="editor-comment"><p>13. Are the authors willing to open the BashtheBug platform so that other labs – especially in developing countries – can provide datasets?</p></disp-quote><p>Yes, but it depends on what you mean by open! At its heart BashTheBug is a project on the Zooniverse platform which has admin users who can modify tasks, upload images etc. If it wasn’t too much work, we could refactor the project to match their 96-well plate and upload the images – this should not be too hard since the software (AMyGDA) which segments the photographs of the 96-well plates is designed to accept different plate designs. If we didn’t have the time, the alternative would be to make them admin users and then they could upload their own images and retrieve the results, thereby making use of the BashTheBug brand and volunteer base.</p><disp-quote content-type="editor-comment"><p>14. The mention that CRyPTIC is seeking to expand knowledge of genetic variants of TB and the reasons for this are very exciting. However, this sentiment is not echoed again in the manuscript. Some elaboration of this, in the context of your findings, might greatly strengthen your discussion, if you feel it is appropriate. For example, based on what the volunteers can do, what work do you plan to do based on it, have you learned anything new about the different genomes, and do you have any recommendations for other projects sequencing microbial genomes?</p></disp-quote><p>We agree – the discussion about CRyPTIC in the Introduction felt misplaced as it left the reader “hanging” so we have moved it to the second paragraph of the Discussion and expanded upon it, including citing several of the early primary research outputs from the project as well as some personal thoughts on the experience, as researchers, of setting up and running a citizen science project.</p><disp-quote content-type="editor-comment"><p>15. It would have been extremely interesting, if at all possible, to see what would have happened if several experts had classified the same plate – what might the errors have been then? You note that experts are inconsistent, but it might be interesting to calculate how much so, compared to volunteers. Can you comment on this?</p></disp-quote><p>The CRyPTIC project did just this in a previous publication</p><p>(https://doi.org/10.1128/AAC.00344-18). The main aim of the paper was to validate the UKMYC5 plate design and determine the optimal reading methods for a laboratory scientist and also the optimal incubation time. Each plate was read by two scientists in each laboratory and hence the inter-reader agreement was measured as a function of incubation time (Figure 2B in the above reference). We have added a sentence to the Results comparing the reproducibility of the volunteers to this existing publication.</p><disp-quote content-type="editor-comment"><p>16. Please explain why the intervals of 7, 10, 14 and 21 days of incubation were chosen for the analysis of the cultures? (The paper does not mention that TB cultures grow unusually slowly, and that of course this is an additional reason why treatment is difficult). Readers working in the field will doubtless be aware of this fact, but those in citizen science or other disciplines may not be</p></disp-quote><p>The reviewer is correct that we’ve assumed a reader will know that TB cultures grow unusually slowly so have inserted a sentence to the Introduction to make this point. In addition (and this is related to §2.18) we have also rewritten the Image Dataset section in the Methods to make it clearer that the images came from the earlier study which was setup to determine the optimal incubation period, and it used 7, 10, 14 and 21 days.</p><disp-quote content-type="editor-comment"><p>17. Can you describe where the 20 000 cultures came from?</p></disp-quote><p>Fortunately, the main CRyPTIC analysis papers have been pre-printed (with a few in press) since this manuscript was submitted to <italic>eLife</italic> and we can therefore cite these to explain in more detail how the 20,637 samples were collected, which countries they originated from, the mix of lineages, the prevalence of resistance etc. We have rewritten the first paragraph of the Results, including citing two of the CRyPTIC studies to provide more context.</p><disp-quote content-type="editor-comment"><p>18. You describe, "Thirty one vials containing 19 external quality assessment (EQA) M. tuberculosis strains, including the reference strain H37Rv ATCC 272948", which "were sent to seven participating laboratories". How were these selected? Please add a summary of how this process took place. In the text, this paragraph is confusing because seven separate institutions were mentioned but only two members of staff – did these staff travel between institutions?</p></disp-quote><p>The EQA dataset was constructed early in the CRyPTIC project and hence the seven participating laboratories were simply the first to join the consortium. We were reluctant to spend too long describing how the dataset is constructed since that is described in detail in the citation but have added some text explaining a little more how the dataset was built and clarifying that the two members of staff were from each laboratory.</p><disp-quote content-type="editor-comment"><p>19. The paper would be clearer if there had been a separate section specifically on comparing Volunteer measurements with Expert and/or with Expert+AMyGDA.</p></disp-quote><p>We considered this when writing the manuscript but instead decided we should primarily consider the dataset that is likely to have fewer errors (Expert+AMyGDA) but acknowledge its possible problems and then after each analysis, refer to the Expert dataset as well to check that the conclusion still holds.</p><disp-quote content-type="editor-comment"><p>20. Figure 1 (A, B and C) is too small.</p></disp-quote><p>We have made it larger.</p><disp-quote content-type="editor-comment"><p>21. It would be very useful to explain a little more fully exactly why "This is slightly less than the 95% threshold set by ISO for aerobic bacteria and therefore the volunteers do not need this criterion". The sentence implies the opposite, given that the volunteers do not meet the 95% threshold, this is exactly why they would need this criterion. Some clarification here might strengthen the main finding, that a crowd of volunteers can perform the required task.</p></disp-quote><p>The volunteers did not meet the 95% threshold for reproducibility, and we are straightforwardly acknowledging that here. It is, however, a little nuanced since there is no ISO standard for Mycobacterial AST devices and so we are applying the standard for aerobic bacteria. We have reworded the sentence slightly.</p><disp-quote content-type="editor-comment"><p>22. It may possibly be worth quoting or referencing some literature on "the wisdom of crowds" or some such topic to explain more fully why you chose citizen science, and a large number of untrained eyes rather than a small number of trained ones, to analyse your dataset.</p></disp-quote><p>Thanks for drawing our attention to this; we have added a sentence in the concluding paragraph of the Introduction referring to Francis Galton’s Nature letter <italic>Vox Populi</italic> in 1907 which appears to be a plausible originator of this idea.</p><disp-quote content-type="editor-comment"><p>23. The authors mention several biases, e.g. the different error sources in the "expert" versus the "AMyGDA" analyses, and in the tendency of volunteers to pick high MICs while the AMyGDA picks low ones, but overall throughout the paper it is less clear how these were actually corrected for or otherwise how these bias were addressed.</p></disp-quote><p>We have broken out the section discussing biases into its own paragraph in the Discussion and have stated that CRyPTIC at present ignores these biases, i.e. it assumes that if MICs measured by AMyGDA and BashTheBug agree then the measurement is correct. We have also made the point that taking the biases into account would likely reduce the measurement error yet further as well as increasing the number of samples</p><p>where two or more measurement methods agree (since agree would be more nuanced than “has the same MIC”).</p></body></sub-article></article>