Functional Dark Matter — Experimentally Prioritized Novel Genetic Systems

Nearly one in four bacterial genes lacks functional annotation ("hypothetical protein"), yet many have experimentally measured fitness effects in the Fitness Browser's 27M measurements across 7,552 conditions. Previous observatory projects have already: predicted function for 6,691 hypothetical proteins via ICA modules (fitness_modules), identified 1,382 hypothetical essentials (essential_genome), and linked 177,863 FB genes to pangenome conservation status (conservation_vs_fitness).

This project builds on those foundations with genuinely new analyses: (1) GapMind pathway gap-filling to find dark genes encoding missing enzymatic steps, (2) cross-organism fitness concordance testing whether orthologs of the same dark gene show the same phenotypes, (3) biogeographic analysis using AlphaEarth satellite embeddings and NCBI metadata to test whether carrier environments match lab fitness conditions, and (4) integrated experimental prioritization producing a ranked candidate list.

Step 1: Cataloging the dark gene landscape (NB01)

Before prioritizing genes for experiments, we need to know what we're working with. The census integrates four prior observatory projects (fitness_modules, essential_genome, conservation_vs_fitness, module_conservation) with direct Fitness Browser queries to build a single table of every dark gene across all 48 organisms. This integration is necessary because prior projects answered different questions and stored results in different formats — no unified catalog existed.

The census identifies 57,011 dark genes (24.9% of 228,709 total), of which 39,532 (69.3%) link to the pangenome, 6,142 (10.8%) belong to ICA fitness modules with function predictions, 7,787 (13.7%) show strong fitness effects (|fit| ≥ 2), and 9,557 (16.8%) are essential. The intersection that matters most for biogeographic analysis — accessory dark genes with strong fitness — is 511 genes.

Conclusion: One quarter of bacterial genes remain dark, but the problem is tractable — 17,344 have measurable phenotypes (fitness or essentiality) and 39,532 connect to the pangenome. The unified catalog enables all downstream analyses.

Data: data/dark_genes_integrated.tsv (228,709 rows, all genes with cross-references), data/dark_genes_only.tsv (57,011 dark gene subset). Notebook: 01_integration_census.ipynb.

Step 2: Adding new inference layers (NB02)

The prior projects characterized dark genes through fitness, conservation, and co-regulation. Three additional inference methods were needed to fill gaps in that picture:

GapMind pathway gap-filling asks: do dark genes encode missing enzymatic steps in nearly-complete metabolic pathways? If a species' amino acid biosynthesis pathway is 90% complete and a dark gene sits near the gap, that gene becomes a candidate for the missing enzyme. Querying 305M GapMind pathway rows (filtered to 44 FB-linked species) identified 1,256 organism-pathway pairs where dark genes co-occur with pathway gaps — providing metabolic context that fitness data alone cannot.

Cross-organism fitness concordance asks: when orthologs of the same dark gene exist in multiple organisms, do they show fitness effects under the same conditions? If a dark gene family matters for nitrogen metabolism in both MR-1 and E. coli, that's stronger evidence than a single-organism observation. Testing 65 ortholog groups present in 3+ FB organisms, motility-related dark genes showed the strongest concordance — consistent with conserved but incompletely annotated chemotaxis machinery.

Phylogenetic breadth asks: how widespread is each dark gene family across the full pangenome (27,690 species, not just the 48 FB organisms)? A dark gene conserved across 5 phyla is more likely to encode a fundamental function than one restricted to a single clade. Mapping 30,756 gene clusters to taxonomy revealed that species count (1–33) provides finer resolution than the coarse universal/clade-restricted classification.

Conclusion: Each inference layer adds a dimension that fitness data alone cannot provide. GapMind places dark genes in metabolic context (1,256 candidate gap-fillers). Concordance confirms that dark gene phenotypes are reproducible across organisms, not single-species artifacts (65 families tested; motility genes strongest). Phylogenetic breadth distinguishes broadly conserved dark genes (likely fundamental) from clade-restricted ones (likely niche-specific).

Data: data/gapmind_gap_candidates.tsv (1,256 organism-pathway pairs), data/concordance_scores.tsv (65 OG concordance scores), data/phylogenetic_breadth.tsv (30,756 clusters with taxonomic breadth). Notebook: 02_gapmind_concordance_phylo.ipynb.

Step 3: Testing whether lab phenotypes match nature (NB03–NB04)

Fitness data tells us what dark genes do in the lab. Biogeographic analysis tests whether those lab phenotypes are ecologically relevant — whether genomes carrying stress-responsive dark genes actually come from stressful environments. This matters because a gene that matters both in the lab and in nature is a better experimental target than one with only a lab phenotype.

Within-species carrier vs. non-carrier tests (controlling for phylogeny) found 10/137 clusters with significant environmental enrichment (FDR < 0.05) and 1/67 with AlphaEarth embedding divergence. The directional lab-field concordance rate was 61.7% (29/47 testable clusters), exceeding chance. NMDC independent validation confirmed all 4 pre-registered predictions (nitrogen carriers correlate with nitrogen-rich environments, pH carriers with pH-extreme environments, etc.) and 76/105 abiotic correlations reached significance. While the NMDC signal is inflated by compositional coupling (see Limitations), the directional concordance across independent datasets supports the inference that lab fitness phenotypes reflect real ecological function.

Conclusion: Lab fitness phenotypes are not lab artifacts — they correspond to real environmental selection pressures. The 61.7% directional concordance rate and 4/4 confirmed NMDC predictions mean that genes important under stress in the lab tend to come from organisms found in stressful environments. This validates fitness data as a proxy for ecological function and strengthens the case for prioritizing genes with both lab and field signals.

Data: data/biogeographic_profiles.tsv (31 species-level profiles), data/carrier_noncarrier_tests.tsv (151 within-species tests), data/lab_field_concordance.tsv (47 pre-registered concordance tests), data/nmdc_validation.tsv (105 NMDC abiotic correlations). Notebooks: 03_biogeographic_analysis.ipynb, 04_lab_field_concordance.ipynb.

Step 4: Scoring and ranking candidates (NB05)

With fitness importance, conservation, module membership, domain annotations, biogeographic signal, and tractability quantified, a composite score (6 weighted dimensions, each 0–1) ranks all 17,344 dark genes with measurable phenotypes. The purpose of scoring is to translate diverse evidence types into a single prioritization that an experimentalist can act on: the top-ranked genes are those where the most evidence converges and the experimental path is clearest.

The top 100 candidates (scores 0.624–0.715) span 22 organisms, with 82/100 having high-confidence functional hypotheses, 85/100 having module-based predictions, and 97/100 having domain annotations. MR-1 contributes 25/100 top candidates — a consequence of its deep condition coverage (121 conditions) rather than inherent biology.

Conclusion: Multi-dimensional scoring reduces 17,344 phenotype-bearing dark genes to a prioritized list where the top candidates have converging evidence from fitness, conservation, co-regulation, and domain structure. The top 100 are not just statistically interesting — 82% have testable functional hypotheses with specific experimental protocols.

Data: data/scoring_all_dark.tsv (17,344 fully scored genes), data/prioritized_candidates.tsv (top 100 with hypotheses and suggested experiments). Notebook: 05_prioritization_dossiers.ipynb.

Step 5: Robustness and controls (NB06)

Prioritization is only useful if the underlying signals are robust. Three controls were run:

H1b formal test (Fisher's exact, n=7,491): the hypothesis that stress-condition dark genes should be more accessory than carbon/nitrogen genes was rejected (p=0.013, opposite direction: stress genes are 23.0% accessory vs. 25.5% for carbon/nitrogen). This unexpected result reveals that the relationship between condition specificity and pangenome conservation is more complex than a simple stress=accessory, metabolism=core model.

Dark-vs-annotated concordance null (Mann-Whitney, 65 dark vs. 490 annotated OGs): dark genes achieve cross-organism concordance levels indistinguishable from annotated genes (p=0.17). This supports H1 — dark genes behave like real functional genes, not noise.

Scoring weight sensitivity: rank correlations remain high (ρ > 0.93) across 6 alternative weight configurations, but specific top-50 lists are moderately sensitive (64% retention for conservation-dominant weighting). Users should treat rankings as approximate and focus on genes that appear in the top tier across multiple weight schemes.

Conclusion: The prioritization is defensible but not perfect. Dark genes behave statistically like annotated genes (supporting H1), the overall ranking is stable across weight perturbations, and H1b's rejection reveals genuine biological complexity rather than a flaw in the analysis. The main caution: specific rank positions are sensitive to weight choices, so candidates should be evaluated in tiers rather than by exact rank.

Data: data/h1b_test_results.tsv (formal test results), data/annotated_control_concordance.tsv (490 annotated OG scores for null comparison), data/nmdc_trait_validation.tsv (456 trait-condition correlations), data/scoring_sensitivity_nb05.tsv and data/scoring_sensitivity_nb07.tsv (weight sensitivity analyses). Notebook: 06_robustness_checks.ipynb.

Step 6: Essential gene prioritization (NB07)

Essential dark genes (9,557) are invisible to standard RB-TnSeq because no viable mutants exist — they have zero rows in genefitness. They require a separate prioritization using the evidence that is available: gene neighborhood context (what annotated genes sit next to them), domain structure, ortholog conservation, phylogenetic breadth, and CRISPRi tractability. This separate scoring avoids the bias of penalizing essential genes for lacking fitness data they structurally cannot have.

Of 57,011 dark genes, 97.2% have at least one annotated gene within a 5-gene window, and 30,190 (52.9%) have annotated operon partners. The top 50 essential candidates (scores 0.740–0.875) span 15 organisms and all have high-confidence hypotheses derived from neighbor context and domain annotations.

Conclusion: Essential dark genes are the majority (55%) of phenotype-bearing dark genes but are systematically missed by fitness-based scoring. Gene neighborhood analysis recovers functional context for nearly all of them (97.2% have annotated neighbors), and CRISPRi provides an experimental path that transposon mutagenesis cannot. The top 50 essential candidates have scores (0.740–0.875) that exceed the fitness-active top 100 (0.624–0.715), reflecting stronger conservation and neighborhood signals.

Data: data/gene_neighbor_context.tsv (57,011 neighbor profiles), data/essential_dark_scored.tsv (9,557 scored essentials), data/essential_prioritized_candidates.tsv (top 50 with CRISPRi experiment designs). Notebook: 07_essential_dark_prioritization.ipynb.

Step 7: Synteny and co-fitness validation (NB08)

The NB07 operon predictions use a single-genome positional heuristic. NB08 adds two independent validation layers:

Cross-species synteny asks: is the dark-gene/annotated-gene neighborhood conserved across multiple organisms? Testing 21,011 pairs across 48 FB genomes, conserved neighborhoods strengthen the functional inference (if the same two genes sit together in 10 species, the association is unlikely to be accidental).

Co-fitness validation asks: do predicted operon partners show correlated fitness profiles across conditions? Testing 32,075 pairs, co-fitness-confirmed operons provide the strongest functional inference for essential genes that lack direct fitness data.

Re-scoring all 17,344 fitness-active and 9,557 essential dark genes with synteny and co-fitness evidence produced 300 improved candidates (200 fitness-active + 100 essential) with evidence-weighted experimental recommendations.

Conclusion: Cross-species synteny and co-fitness provide independent validation of operon-based functional inferences. Genes whose neighborhood is conserved across multiple organisms and whose operon partners show correlated fitness profiles have the strongest evidence for guilt-by-association function prediction. The 300 improved candidates integrate all evidence layers accumulated across the project.

Data: data/conserved_neighborhoods.tsv (21,011 synteny-scored pairs), data/cofit_validated_operons.tsv (32,075 cofit-scored pairs), data/improved_candidates.tsv (300 re-scored candidates), data/experimental_roadmap.tsv (30 organism experiment priorities). Notebook: 08_improved_neighborhoods.ipynb.

Step 8: Final synthesis — darkness spectrum, covering set, action plan (NB09)

The preceding analyses produced gene-level evidence and organism-level candidates, but an experimentalist still cannot answer: how many organisms do I need to study, and which ones? The final synthesis translates gene-level priorities into an experimental campaign.

Darkness spectrum: All 57,011 dark genes classified by evidence depth — T1 Void (4,273, 7.5%, zero evidence), T2 Twilight (12,282, 21.5%, one clue), T3 Dusk (16,103, 28.2%, two converging hints), T4 Penumbra (22,500, 39.5%, 3–4 evidence lines), T5 Dawn (1,853, 3.3%, nearly characterized). Only 7.5% are truly unknown; the majority have substantial evidence and need targeted experiments, not broad screens.

Minimum covering set: A greedy weighted set-cover algorithm (optimizing priority value × tractability × phylogenetic diversity) selects 42 organisms (28 genera) covering 95% of total priority. MR-1 ranks first. 32 organisms achieve 80% coverage. Each gene is assigned to exactly one organism for experimental follow-up.

Action plan: 14,450 genes are classified as hypothesis-bearing (with specific condition recommendations from fitness data, module prediction, or neighbor context); 2,038 are classified as darkest (requiring broad phenotypic screens). 8,900 essential genes in the covering set are recommended for CRISPRi approaches.

Conclusion: The darkness spectrum reveals that the "dark matter" problem is not monolithic — most dark genes (92.5%) have at least some evidence, and 39.5% have 3–4 converging lines. The evidence-weighted set-cover algorithm translates gene priorities into a practical experimental campaign: 42 organisms covering 95% of all scored dark gene priority. A complementary conservation-weighted covering set (NB11, Finding 14) provides an alternative organism ordering optimized for discovering functions of broadly conserved true knowledge gaps.

Data: data/dark_gene_census_full.tsv (57,011 genes with darkness tier and evidence flags), data/minimum_covering_set.tsv (16,488 gene-to-organism assignments), data/experimental_action_plan.tsv (42 organism action plans). Notebook: 09_final_synthesis.ipynb.

Step 9: Pangenome-scale conservation × hypothesis classification (NB11, NB11b)

The NB09 evidence-weighted approach relies on Fitness Browser data, which limits conservation assessment to the 48 FB organisms. The eggNOG breadth classification used in NB05's s_pangenome score is non-discriminative: 99.9% of dark gene clusters map to "universal" root-level OGs, producing identical conservation scores. NB11 addresses this by querying the full 27,690-species GTDB r214 pangenome.

Full pangenome conservation (NB11b): A Spark query explodes eggNOG_OGs annotations across 93.5M gene clusters, matching all comma-separated OG entries against 11,774 dark gene root OGs. Species counts now range from 1 to 27,482 (median 135, mean 2,128) — replacing the previous 1-to-33 range from FB-only analysis. OG_id propagation recovers 5,206 additional dark genes by transferring root_og assignments from genes with known pangenome links to genes in the same 48-organism ortholog group.

Taxonomic tier × hypothesis classification: Each dark gene OG is classified into 8 taxonomic tiers (kingdom 55.9%, class 11.0%, family 10.5%, genus 6.9%, mobile 6.5%, phylum 4.8%, order 3.9%, species 0.5%) and 3 hypothesis status tiers (strong testable hypothesis 6.0%, weak lead 52.5%, true knowledge gap 41.5%). The importance score = (tier-adjusted conservation + log₂-scaled species fraction) × ignorance multiplier ranks all OGs, with kingdom-level true knowledge gaps at the top.

Conservation-weighted covering set: A greedy set-cover algorithm optimizing Σ(importance) × tractability × phylo_bonus selects 42 organisms (28 genera) covering 95.6% of importance-weighted priority. S. meliloti ranks first (1,630 OGs, 195 kingdom gaps), followed by P. putida (1,043 OGs, 172 kingdom gaps) and MR-1 (805 OGs, 105 kingdom gaps). Per-organism experimental plans specify recommended approaches by tier × hypothesis status.

Extended covering set (NB11c): A curated list of 73 organisms (48 FB + 25 literature-curated with TnSeq/CRISPRi resources from Bacillota, Actinomycetota, and Campylobacterota) was assembled to address the Proteobacteria-heavy FB sampling bias. A Spark query mapped genus-level OG membership for 25 non-FB organisms across the pangenome (24 genera, 53,970 genus-OG pairs). Running the covering set with all 73 candidates produces a 50-organism set covering 98.7% of OGs across 6 phyla (vs. 41 organisms, 4 phyla for FB-only), with 16 non-FB organisms selected including P. aeruginosa PAO1 (#1, 3,713 OGs), M. tuberculosis (#6, Actinomycetota), and C. jejuni (#48, Campylobacterota).

Conclusion: The full pangenome reveals that 55.9% of dark gene OGs are kingdom-level — present across multiple phyla and thousands of species — demonstrating that functional dark matter is not a minor annotation gap but a fundamental limitation in understanding conserved biology. The conservation-weighted covering set provides an alternative experimental ordering optimized for discovering completely unknown functions, complementing the evidence-weighted ordering from NB09.

Data: data/og_pangenome_distribution.tsv (11,774 OGs with full pangenome counts), data/dark_gene_classes.tsv (57,011 genes with tier/status/importance), data/og_importance_ranked.tsv (11,774 OGs ranked by importance), data/conservation_covering_set.tsv (42 FB-only organisms), data/conservation_experiment_plans.tsv (42 organism plans), data/extended_tractable_organisms.tsv (73 organisms), data/non_fb_genus_og_coverage.tsv (53,970 genus-OG pairs), data/extended_covering_set.tsv (50-organism extended covering set). Notebooks: 11_conservation_classes.ipynb, 11b_extended_conservation.ipynb, 11c_extended_covering_set.ipynb.

Hypothesis Assessment

H1 is partially supported; H0 can be partially rejected. Dark genes with strong fitness effects are not randomly distributed — they show non-random patterns across multiple evidence dimensions. Critically, a matched comparison of dark vs. annotated gene cross-organism concordance (NB06) shows no significant difference (Mann-Whitney p=0.17, KS p=1.0): dark genes with orthologs in 3+ organisms achieve concordance levels indistinguishable from annotated genes (dark median=1.0, annotated median=1.0; dark mean=0.976, annotated mean=0.985). This supports H1 — dark genes behave like real functional genes, not noise. The specific sub-hypothesis assessments:

• H1a (Functional coherence): Supported. 6,142 dark genes co-regulate with annotated genes in ICA modules, and 85/100 top candidates have module-based function predictions. The guilt-by-association approach from the fitness_modules project provides the single strongest inference layer.

• H1b (Conservation signal): Not supported. Formal testing (Fisher's exact, n=7,491 dark genes; NB06) showed the opposite of the predicted pattern: stress-condition dark genes are 23.0% accessory, while carbon/nitrogen dark genes are 25.5% accessory (OR=1.15, p=0.013). The hypothesis that stress genes should be more accessory than carbon/nitrogen genes is rejected. This suggests that the relationship between condition specificity and pangenome conservation is more complex than a simple stress=accessory, metabolism=core dichotomy.

• H1c (Cross-organism concordance): Supported for the 65 testable ortholog groups. Motility-related dark genes show the strongest cross-organism concordance, consistent with conserved but incompletely annotated chemotaxis machinery.

• H1d (Biogeographic pattern): Supported. 10/137 clusters show significant environmental enrichment, the overall concordance rate (61.7%) exceeds the 50% chance level (binomial p = 0.072, Fisher's combined p = 0.031; NB10), and NMDC independent validation confirmed all 4 testable pre-registered predictions (nitrogen~nitrogen, pH~pH, anaerobic~dissolved oxygen). The strongest within-species signals are in Pseudomonas and P. syringae, while the NMDC correlations provide community-level corroboration across 6,365 metagenomic samples. Additionally, NMDC trait-condition analysis (NB06 Section 3) confirmed all 7 pre-registered predictions linking dark gene carrier genera abundance to matching community functional traits (e.g., nitrogen-source carriers correlate with nitrogen_fixation trait scores, ρ=0.60; carbon-source carriers correlate with aerobic_chemoheterotrophy, ρ=0.73). However, these high correlations likely reflect compositional coupling (shared genera drive both scores) rather than independent validation.

• H1e (Pathway integration): Suggestive with partial domain-level support. GapMind identifies 1,256 organism-pathway pairs where dark genes co-occur with metabolic gaps. NB10 provides curated enzymatic domain matching: of 57,011 dark genes, 3,186 have annotations (EC numbers, PFam domains, or functional keywords) compatible with at least one gapped pathway, yielding 42,239 gene-pathway candidates including 5,398 high-confidence EC prefix matches. These domain-compatible candidates narrow the search from organism-level co-occurrence to enzymatically plausible gap-fillers. Full confirmation of specific gene-to-step assignments requires AlphaFold structure prediction or experimental enzymology.

Literature Context

The 24.9% dark gene fraction aligns with published estimates of 25–40% hypothetical genes in typical bacterial genomes (Makarova et al. 2019, Biochem Soc Trans). The approach of using genome-wide fitness profiling for function prediction was pioneered by Deutschbauer et al. (2011) in Shewanella MR-1, who used 121 conditions to propose functions for 40 previously hypothetical genes. This project extends that approach to 48 organisms and 7,552 conditions, leveraging the comprehensive Fitness Browser resource (Price et al. 2018, Nature).

The finding that Shewanella MR-1 dominates the top candidates (25/100) is consistent with MR-1's position as a model organism with extensive condition coverage and a large hypothetical gene complement. Vaccaro et al. (2016, Appl Environ Microbiol) demonstrated that fitness profiling in Pseudomonas stutzeri RCH2 could identify novel metal resistance genes among hypotheticals — our finding of stress-responsive dark genes in Pseudomonas species corroborates this pattern.

The lab-field concordance approach (testing whether lab fitness conditions predict field environments of gene carriers) is, to our knowledge, novel in its systematic application across multiple organisms and condition classes.

Recent large-scale efforts to address the functional dark matter problem provide context for the scale of our contribution. Pavlopoulos et al. (2023, Nature) identified 106,198 novel protein families from 26,931 metagenomes, doubling the number of known protein families — demonstrating that the dark proteome is vast even beyond cultivated organisms. Zhang et al. (2025, Nature Biotechnology) developed FUGAsseM, which predicts high-confidence functions for >443,000 protein families from microbial communities using metatranscriptomic coexpression. Our approach is complementary: while those methods operate at metagenomic scale with computational prediction, we provide experimentally grounded prioritization — every gene in our catalog has measured fitness phenotypes from controlled lab experiments, and the top candidates come with specific experimental protocols. The darkness spectrum classification (T1 Void through T5 Dawn) provides a structured framework absent from prior metagenomic surveys, where uncharacterized proteins are typically treated as a single class.

Alam et al. (2011, BMC Res Notes) prioritized orphan proteins in Streptomyces coelicolor using phylogenomics and gene expression, demonstrating that multi-evidence integration can rank unknowns even in a single organism. Our approach extends this to 48 organisms with 6 evidence dimensions, and adds the set-cover organism selection step that translates gene-level priorities into a concrete experimental campaign.

Novel Contribution

This project contributes:

1. A unified dark gene catalog (57,011 genes across 48 bacteria) integrating fitness, conservation, module, ortholog, and domain data from 4 prior observatory projects — previously fragmented across separate analyses.

2. Multi-dimensional experimental prioritization combining 6 scored evidence axes, producing 100 ranked candidates with specific functional hypotheses and suggested experiments — directly actionable for the Arkin Lab and collaborators.

3. Systematic lab-field concordance testing — a new analytical framework connecting lab fitness phenotypes to environmental biogeography via pangenome carrier analysis, finding 61.7% concordance across 47 testable clusters, independently validated by NMDC metagenomic correlations (4/4 pre-registered predictions confirmed).

4. Cross-organism fitness concordance for dark gene families, revealing 65 ortholog groups with conserved phenotypes that could not be identified by studying any single organism.

5. Darkness spectrum classification — a five-tier evidence inventory (T1 Void through T5 Dawn) that for the first time quantifies "how dark" each gene is across 6 independent evidence axes. This reveals that only 7.5% of dark genes (4,273 T1 Void) are truly unknown with zero evidence; the majority (39.5%, T4 Penumbra) have 3–4 converging lines of evidence and are ripe for targeted characterization. This framework enables resource allocation: T5 Dawn genes need confirmation, T4 Penumbra genes need targeted experiments, T1 Void genes need broad screens.

6. Dual-route covering set optimization — two complementary greedy weighted set-cover algorithms: Route A (evidence-weighted, NB09) selects 42 organisms optimized for genes with testable hypotheses; Route B (conservation-weighted, NB11) selects 42 organisms optimized for discovering functions of broadly conserved true knowledge gaps. The routes share 39 organisms but produce different orderings reflecting different experimental strategies. An extended tractable organism list (73 organisms: 48 FB + 25 literature-curated from 3 additional phyla) provides a pathway to expand experimental coverage beyond the Proteobacteria-dominated Fitness Browser.

Limitations

1. Environmental metadata sparsity: AlphaEarth embeddings cover only 28% of genomes (83K/293K), and NCBI isolation source metadata is inconsistent, limiting the power of biogeographic tests.

2. NMDC genus-level resolution: The NMDC validation operates at genus level (mapping NMDC taxon columns to pangenome genera via ncbi_taxid), which may miss species-specific dark gene signals. Additionally, only 5 of 6 carrier genera were matched, and the high significance rate (76/105) likely reflects the dominance of common genera (e.g., Pseudomonas, Klebsiella) in both datasets.

3. Annotation bias: Some "hypothetical" genes may have annotations in databases not checked (UniProt, InterPro, recent NCBI updates). The dark gene count (57,011) likely overestimates the true number of functionally uncharacterized genes.

4. Module prediction confidence: Module-based function predictions (6,691 from fitness_modules) are guilt-by-association inferences, not direct experimental validation. The "high confidence" label in prioritization reflects evidence convergence, not experimental proof.

5. Condition coverage unevenness: Not all 48 organisms were tested under the same conditions. Organisms with more conditions (e.g., MR-1 with 121) produce more specific phenotypes, biasing them toward higher prioritization scores.

6. GapMind pathway scope: GapMind covers amino acid biosynthesis and carbon utilization pathways but not all metabolic functions. Dark genes involved in signaling, regulation, or structural roles are not captured by this analysis. Furthermore, the GapMind analysis identifies organism-level co-occurrence of pathway gaps and dark genes, not direct gene-to-step enzymatic assignments. NB10 partially addresses this with curated pathway-enzyme domain matching (EC prefix, PFam family, and keyword compatibility), identifying 42,239 domain-compatible gene-pathway candidates including 5,398 high-confidence EC matches. However, full gene-to-step validation would require AlphaFold structure prediction or experimental enzymology.

7. Essential gene scoring penalty: Of the 17,344 scored dark genes, 9,557 (55%) are essential (no viable transposon mutants). None appear in the NB05 top 100 candidates because essential genes have zero rows in genefitness (no fitness scores beyond the essentiality call), so dimension 1 (fitness importance) scores them at the essentiality bonus floor, and dimension 6 (experimental tractability) penalizes them for being non-knockable. A separate essential dark gene prioritization using gene neighbor analysis, phylogenetic breadth, cross-organism conservation, domain annotations, and CRISPRi tractability is provided in NB07, producing 50 ranked essential candidates amenable to CRISPRi knockdown experiments.

8. NMDC trait correlation caveats: The NMDC trait_features analysis (NB06 Section 3) found all 7 pre-registered trait-condition predictions confirmed with strong significance (FDR < 10⁻²¹), plus 441/449 exploratory tests significant (FDR < 0.05). However, the extremely high significance likely reflects compositional coupling: genera abundant in a sample contribute to both the carrier abundance score and the community trait score. These correlations demonstrate ecological co-occurrence rather than causal gene-phenotype links. NB10 provides partial null approximations: (1) a sign test on 7/7 pre-registered trait directions (p = 0.0078) and 4/4 pre-registered abiotic directions (p = 0.0625) confirms non-random directionality; (2) Fisher's combined probability across all 11 pre-registered tests yields p ≈ 0; (3) an inflation factor of ~20× for exploratory tests (441/449 significant vs. 22.5 expected) quantifies the compositional coupling. A full permutation test shuffling sample labels remains a future direction requiring Spark access to raw per-sample matrices.

9. Dark-vs-annotated controls partial: The concordance null control (NB06) shows dark genes achieve concordance levels indistinguishable from annotated genes, supporting H1. However, equivalent null controls were not run for the biogeographic and lab-field concordance analyses. NB10 provides a binomial test (29/47 vs. p = 0.5, one-sided p = 0.072) and Wilson score 95% CI [0.474, 0.742], showing the concordance rate is marginally above chance. Fisher's combined probability across all 47 individual tests yields p = 0.031. A full null comparison using annotated accessory genes through the same biogeographic pipeline would further strengthen the H0 rejection but requires Spark for carrier genome queries.

1. Gene neighborhood methodology gap: NB07's operon predictions use a single-genome positional heuristic (5-gene window, same strand, ≤300 bp gap). NB08 adds cross-species synteny validation across 48 Fitness Browser genomes and co-fitness confirmation, but still falls short of industry-standard tools. DOOR v2.0 uses intergenic distance, neighborhood conservation, phylogenetic distance, and experimentally validated training sets across thousands of genomes. STRING v12 combines genomic neighborhood, gene fusion, phylogenetic co-occurrence, co-expression, and text mining across >14,000 organisms. EFI-GNT provides interactive genome neighborhood diagrams across all sequenced genomes with SSN-guided filtering. Our analysis captures the most accessible signal (positional heuristic + limited cross-species synteny + co-fitness) but lacks gene fusion detection, regulatory element analysis, and broad taxonomic sampling that would improve prediction accuracy.

2. Scoring weight sensitivity: Both prioritization frameworks use arbitrary expert-assigned weights. Sensitivity analysis (NB05/NB07) shows that while overall rank correlations remain high across alternative weight configurations (ρ > 0.93), the top-ranked candidate lists are moderately sensitive to weight choices. For NB05 (fitness-active): conservation-dominant and drop-tractability configurations each retain only 32/50 original top candidates (64%). For NB07 (essential): dropping tractability retains only 18/50 (36%) and dropping neighbor context retains 24/50 (48%). NB10 provides robust rank indicators: across all 6 fitness-active weight configurations, 18 genes remain in the top 50 and 35 remain in the top 100 regardless of weight choice. For essential genes, 6 remain always-top-50 and 19 always-top-100. These "always-top" candidates are the most defensible targets for experimental follow-up. Full per-gene rank ranges are in data/robust_ranks_fitness.tsv and data/robust_ranks_essential.tsv.

1. Proteobacteria-dominated organism set: The 48 Fitness Browser organisms are 77% Pseudomonadota (37/48). The extended covering set (NB11c) partially addresses this by incorporating 25 non-FB organisms from Bacillota, Actinomycetota, and Campylobacterota, expanding phylum coverage from 4 to 6. However, the non-FB OG coverage is estimated at the genus level (any species in the genus has the OG), which overestimates individual organism coverage. Additionally, non-FB organisms lack Fitness Browser condition profiling, so their dark genes cannot be assigned to condition-specific experiments — only broad phenotypic screens. Bacillota organisms (B. subtilis, S. aureus) were not selected by the covering set algorithm because their OGs are subsets of coverage provided by Pseudomonadota, despite their value for studying genes in native Gram-positive contexts.

1. Execute the dual-route experimental campaign — Route A (NB09, evidence-weighted) identifies 42 organisms covering 95% of composite priority; start with MR-1 stress/nitrogen screens and E. coli CRISPRi for targeted hypothesis testing. Route B (NB11, conservation-weighted) identifies 42 organisms covering 95.6% of importance-weighted priority; start with S. meliloti, P. putida, and MR-1 broad phenotypic screens for novel function discovery. The two routes share 39 organisms and are complementary: Route A produces mechanistic insights for genes with hypotheses; Route B discovers functions for broadly conserved true knowledge gaps.

2. Targeted validation of top candidates — the top 5 fitness-active candidates (4 in MR-1, 1 in P. putida N2C3) are immediately testable via RB-TnSeq under their predicted condition classes. Specific targets: AO356_11255 (D-alanyl-D-alanine carboxypeptidase prediction, EamA domain — test under nitrogen limitation); MR-1 202463 (YGGT domain, T5 Dawn, composite score 0.698 — test under multiple stress conditions); MR-1 199738/203545/202450 (K03306 paralog family — test under nitrogen limitation and compare single/double mutants).

3. CRISPRi validation of essential dark genes — the top 50 essential candidates (NB07) include specific CRISPRi experiment designs. The highest-priority targets are Keio:14796 (YbeY domain, score 0.875, predicted ion transport), MR-1:200382 (RimP_N/DUF150_C, score 0.874, predicted ribosome assembly), and Koxy:BWI76_RS08540 (OmpA/TIGR02802, score 0.865, predicted cell division). The covering set assigns 8,900 essential genes to specific organisms for CRISPRi-based approaches.

4. Protein structure prediction — for the top 100 candidates and especially the 1,853 T5 Dawn genes (nearly characterized), AlphaFold2 structure predictions could confirm functional hypotheses and distinguish between DUF families. The 4,273 T1 Void genes, which lack all evidence, would particularly benefit from structure-based function prediction as the only remaining computational inference strategy.

5. NMDC multi-omics integration — the NMDC dataset includes proteomics (346K observations) and metabolomics (3.1M observations) that were not used here. Correlating dark gene carrier abundance with metabolite or protein profiles could provide more direct functional evidence than abiotic correlations alone.

6. Expand the covering set beyond Fitness Browser organisms — the current covering sets draw only from 48 FB organisms, which are 77% Pseudomonadota. An extended tractable organism list (data/extended_tractable_organisms.tsv) curates 25 additional organisms with published TnSeq/CRISPRi resources from Bacillota (6), Actinomycetota (2), Campylobacterota (2), and additional Pseudomonadota (15). Incorporating these into Route B's conservation-weighted covering set would enable experimental characterization of kingdom-level OGs (55.9% of dark gene OGs) in their native Gram-positive, Actinobacterial, and Campylobacterial contexts. This requires a pangenome Spark query to map each organism's genes to dark gene root_ogs — an extension of the NB11b pipeline.

7. Community resource — publish the darkness spectrum census (dark_gene_census_full.tsv) and experimental action plan as a community resource for bacterial functional genomics, enabling other labs to target specific organisms, tiers, or condition classes matching their expertise and infrastructure.

Sources

Generated Data

• Price MN, Wetmore KM, Waters RJ, Callaghan M, Ray J, Liu H, Kuehl JV, Melnyk RA, Lamson JS, Cai Y, et al. (2018). "Mutant phenotypes for thousands of bacterial genes of unknown function." Nature 557:503–509. PMID: 29769716
• Deutschbauer A, Price MN, Wetmore KM, Shao W, Baumohl JK, Xu Z, Nguyen M, Tamse R, Davis RW, Arkin AP. (2011). "Evidence-based annotation of gene function in Shewanella oneidensis MR-1 using genome-wide fitness profiling across 121 conditions." PLoS Genetics 7:e1002385. PMID: 22125499
• Wetmore KM, Price MN, Waters RJ, Lamson JS, He J, Hoover CA, Blow MJ, Bristow J, Butland G, Arkin AP, Deutschbauer A. (2015). "Rapid quantification of mutant fitness in diverse bacteria by sequencing randomly bar-coded transposons." mBio 6:e00306-15. PMID: 25968644
• Price MN, Deutschbauer AM, Arkin AP. (2024). "A comprehensive update to the Fitness Browser." mSystems 9:e00470-24.
• Vaccaro BJ, Lancaster WA, Thorgersen MP, Zane GM, Younkin AD, Kazakov AE, Wetmore KM, Deutschbauer A, Arkin AP, Novichkov PS, Wall JD, Adams MW. (2016). "Novel Metal Cation Resistance Systems from Mutant Fitness Analysis of Denitrifying Pseudomonas stutzeri." Appl Environ Microbiol 82:6046–6056. PMID: 27474723
• Makarova KS, Wolf YI, Koonin EV. (2019). "Towards functional characterization of archaeal genomic dark matter." Biochem Soc Trans 47:389–398. PMID: 30647141
• Arkin AP, Cottingham RW, Henry CS, Harris NL, Stevens RL, Masber S, et al. (2018). "KBase: The United States Department of Energy Systems Biology Knowledgebase." Nature Biotechnology 36:566–569. PMID: 29979655
• Peters JM, Colavin A, Shi H, Czarny TL, Larson MH, Wong S, Hawkins JS, Lu CHS, Koo BM, Marta E, et al. (2016). "A Comprehensive, CRISPR-based Functional Analysis of Essential Genes in Bacteria." Cell 165:1493–1506. PMID: 27238023
• Peters JM, Koo BM, Patidar R, Heber CC, Tekin S, Cao K, Terber K, Lanze CE, Sirothia IR, Murray HJ, et al. (2019). "Enabling genetic analysis of diverse bacteria with Mobile-CRISPRi." Nature Microbiology 4:244–250. PMID: 30617347
• Tan SZ, Reisch CR, Prather KLJ. (2018). "A robust CRISPRi gene repression system in Pseudomonas." Journal of Bacteriology 200:e00575-17. PMID: 29311275
• Mimee M, Tucker AC, Voigt CA, Lu TK. (2015). "Programming a Human Commensal Bacterium, Bacteroides thetaiotaomicron, to Sense and Respond to Stimuli in the Murine Gut Microbiota." Cell Systems 1:62–71. PMID: 26918244
• Pavlopoulos GA, Baltoumas FA, Liu S, Noval Rivas M, Pinto-Cardoso S, et al. (2023). "Unraveling the functional dark matter through global metagenomics." Nature 622:594–602. PMID: 37821698
• Zhang Y, Bhosle A, Bae S, Franzosa EA, Huttenhower C, et al. (2025). "Predicting functions of uncharacterized gene products from microbial communities." Nature Biotechnology. PMID: 41094150
• Alam MT, Takano E, Breitling R. (2011). "Prioritizing orphan proteins for further study using phylogenomics and gene expression profiles in Streptomyces coelicolor." BMC Research Notes 4:325. PMID: 21899768
• Koo BM, Kritikos G, Farelli JD, Todor H, Tong K, Kimber H, Wapinski I, Galardini M, Caber A, Peters JM, et al. (2017). "Construction and Analysis of Two Genome-Scale Deletion Libraries for Bacillus subtilis." Cell Systems 4:291–305. PMID: 28189581
• Dembek M, Barquist L, Boinett CJ, Cain AK, Mayho M, Lawley TD, Fairweather NF, Fagan RP. (2015). "High-throughput analysis of gene essentiality and sporulation in Clostridium difficile." mBio 6:e02383-14. PMID: 25714712
• de Vries SP, et al. (2017). "Genome-wide fitness analyses of the foodborne pathogen Campylobacter jejuni in in vitro and in vivo models." Scientific Reports 7:1251. PMID: 28455506
• Sastry AV, Gao Y, Szubin R, Hefner Y, Xu S, Kim D, Choudhary KS, Yang L, King ZA, Palsson BO. (2019). "The Escherichia coli transcriptome mostly consists of independently regulated modules." Nature Communications 10:5536. PMID: 31797920

This section distills the entire analysis into an actionable experimental campaign. Two complementary prioritization routes were developed — one optimized for genes with the strongest existing evidence, the other for maximizing discovery of completely unknown functions — and both converge on a tractable set of organisms.

Two complementary prioritization routes

Route A — Evidence-weighted prioritization (NB05–NB09): Scores each dark gene across 6 dimensions (fitness importance 0.25, conservation 0.20, inference 0.20, pangenome 0.15, biogeographic 0.10, tractability 0.10) using data available from the 48 Fitness Browser organisms. This route identifies genes where multiple lines of evidence converge on a testable hypothesis. Best for: targeted experiments on genes where we already know what to test and under what conditions.

Route B — Conservation-weighted prioritization (NB11, NB11b): Queries the full 27,690-species GTDB r214 pangenome to measure taxonomic breadth of each dark gene OG (species range 1–27,482), classifies by taxonomic tier (kingdom through species) and hypothesis status (strong hypothesis, weak lead, true knowledge gap), then ranks by importance = conservation × ignorance. This route identifies broadly conserved genes where we know the least. Best for: discovery experiments targeting the most fundamentally important unknowns in biology.

The two routes produce different organism orderings because they optimize different objectives:

The most valuable dark genes and why

Of 57,011 dark genes across 48 bacteria, 17,344 have experimentally measurable phenotypes — either strong fitness defects (|fit| ≥ 2 in at least one condition) or essentiality (no viable transposon mutants). These are not computationally predicted to matter; they demonstrably matter in the lab. Among them, the convergence of multiple independent evidence lines (fitness phenotype, conservation, co-regulation, domain structure, gene neighborhood, cross-organism concordance) identifies a subset that are both important and interpretable.

Route A top fitness-active candidates — genes where fitness data, module co-regulation, domain annotations, and biogeographic signals converge to produce testable functional hypotheses:

Route B top knowledge gaps — the most broadly conserved genes with zero functional evidence, where experimental characterization would produce the most novel biological insight:

These OGs are among the most universally conserved genes in bacteria yet remain functionally uncharacterized. They are invisible to Route A because they may lack condition-specific fitness effects (they may be essential under all conditions, producing no differential fitness signal).

The top essential dark gene candidates are genes where no viable knockout mutants exist, but gene neighborhood context, cross-species conservation, and domain structure enable CRISPRi knockdown experiments:

These candidates score highest because multiple independent evidence lines converge: they have detectable domains (providing structural clues), belong to widely conserved ortholog groups (confirming biological importance beyond one organism), sit in conserved gene neighborhoods with annotated partners (providing functional context), and are in organisms amenable to CRISPRi (making them experimentally testable despite being essential).

Which organisms and why

Route A organism selection (NB09): A greedy weighted set-cover algorithm selected 42 organisms (28 genera) covering 95% of the total priority value across all scored dark genes. The algorithm optimizes composite priority score × tractability × phylogenetic diversity.

1. Shewanella oneidensis MR-1 (587 genes, tractability 0.8) — Selected first because it combines deep condition coverage (121 conditions historically), a large dark gene complement spanning all tiers (172 T5 Dawn, 257 T4 Penumbra), and high genetic tractability. 544 of its dark genes have specific condition recommendations (stress, nitrogen, carbon). MR-1 is the single highest-impact organism: 25/100 top fitness-active candidates reside here.

2. Pseudomonas fluorescens N1B4 (624 genes, tractability 0.7) — Contributes the most additional uncovered genes after MR-1, particularly in carbon/nitrogen source panels (215 genes) and membrane stress (77 genes).

3. Sinorhizobium meliloti (570 genes, tractability 0.6) — Covers a distinct phylogenetic lineage (Alphaproteobacteria) with 316 T4 Penumbra and 90 T5 Dawn genes.

4. Escherichia coli K-12 (Keio) (368 genes, tractability 0.9) — Highest tractability; 259 hypothesis-bearing genes immediately testable; 205 essential genes ideal for CRISPRi.

5. Klebsiella oxytoca (396 genes, tractability 0.65) — Covers Enterobacteriaceae genes not in E. coli, including top essential candidate BWI76_RS08540 (score 0.865).

Route B organism selection (NB11): A conservation-weighted set-cover algorithm selected 42 organisms covering 95.6% of total importance-weighted priority. The algorithm optimizes Σ(importance) × tractability × phylo_bonus.

1. Sinorhizobium meliloti (1,630 OGs, 195 kingdom gaps, tractability 0.6) — Selected first because its dark genes span the most kingdom-level true knowledge gaps — broadly conserved genes with zero functional evidence.

2. Pseudomonas putida (1,043 OGs, 172 kingdom gaps, tractability 0.8) — High tractability combined with deep coverage of conserved unknowns.

3. Shewanella oneidensis MR-1 (805 OGs, 105 kingdom gaps, tractability 0.8) — Third in Route B vs. first in Route A, because Route B weights conservation breadth over condition-specific evidence depth.

4. Bacteroides thetaiotaomicron (1,382 OGs, 267 kingdom gaps, tractability 0.3) — Highest raw OG count of any organism; deprioritized by low tractability (0.3) but critical for non-Proteobacteria coverage (Bacteroidota).

5. Klebsiella michiganensis (568 OGs, 65 kingdom gaps, tractability 0.65) — Enterobacteriaceae depth beyond E. coli.

The first 10 Route B organisms cover 31% of importance-weighted priority (9,014 genes across 10 genera).

Phylogenetic gap analysis: beyond the 48 Fitness Browser organisms

Both covering sets draw only from the 48 FB organisms, which are heavily biased: 37/48 are Pseudomonadota (Proteobacteria), with Bacteroidota (4), Cyanobacteriota (1), and Halobacteriota (2) the only other phyla represented. Major bacterial phyla — Bacillota (Firmicutes), Actinomycetota (Actinobacteria), and Campylobacterota — have zero FB representation. This means kingdom-level OGs (55.9% of dark gene OGs, present across multiple phyla) cannot be experimentally addressed in their non-Proteobacterial hosts using FB organisms alone.

To address this gap, we curated an extended tractable organism list of 73 organisms (48 FB + 25 literature-curated with published TnSeq or CRISPRi resources; data/extended_tractable_organisms.tsv). The 25 non-FB organisms fill specific phylogenetic gaps:

Incorporating these organisms into the Route B covering set would enable experimental characterization of kingdom-level dark gene OGs in their native Gram-positive, Actinobacterial, and Campylobacterial genomic contexts. This requires a Spark query to map each non-FB organism's genome to dark gene root_ogs via the pangenome — a natural extension of the NB11b pipeline.

How to study them: the experimental strategy

The action plan classifies each dark gene into one of two experimental categories:

Hypothesis-bearing genes (14,450 across all 42 organisms): These have at least one condition recommendation from fitness data, module co-regulation, or gene neighborhood context. The experiment is targeted RB-TnSeq or CRISPRi under the predicted condition. For example, MR-1:202463 (YGGT domain, stress-responsive) should be tested under oxidative, osmotic, and heat stress; the K03306 paralog trio (199738/203545/202450) should be tested under nitrogen limitation with single and double mutant comparisons.

Darkest genes requiring broad screens (2,038 across all 42 organisms): These are T1 Void or T2 Twilight genes with detectable fitness phenotypes but no converging evidence to suggest a specific condition. The experiment is a broad phenotypic screen — RB-TnSeq across a diverse condition panel (the standard Fitness Browser protocol of ~50-100 conditions). For E. coli K-12 alone, 109 genes fall in this category.

Essential genes (8,900 in covering set): These cannot be studied by transposon knockout. The recommended approach is CRISPRi knockdown (Mobile-CRISPRi for non-model organisms, Peters et al. 2019) with growth curves under standard and stress conditions. The top 50 essential candidates have specific CRISPRi experiment designs in NB07.

Prioritized entry points — a three-experiment starting campaign (Route A):

1. MR-1 stress screen: RB-TnSeq under 5 stress conditions (oxidative, osmotic, metal, heat, pH). Addresses 161 hypothesis-bearing dark genes including the #2 overall candidate (202463, fit=6.4 under stress).
2. MR-1 nitrogen screen: RB-TnSeq under nitrogen limitation and amino acid supplements. Addresses the K03306 paralog family (3 genes, fits 3.9–5.5) and 74 total dark genes.
3. E. coli K-12 CRISPRi: Knockdown of top 20 essential dark genes with growth curves. Leverages the highest-tractability organism (0.9) and mature CRISPRi tools. Top target: Keio:14796 (YbeY domain, score 0.875).

Discovery campaign (Route B): For labs focused on novel function discovery rather than hypothesis testing, the Route B ordering — starting with S. meliloti (195 kingdom-level gaps), P. putida (172 kingdom gaps), and MR-1 (105 kingdom gaps) — maximizes the probability of uncovering completely new biology. Broad phenotypic screens under diverse condition panels are recommended for Route B organisms, since the target genes are true knowledge gaps without condition predictions.

These campaigns are complementary: Route A produces mechanistic insights for genes where we have hypotheses; Route B discovers functions for genes where we have none. The full action plans are in data/experimental_action_plan.tsv (Route A) and data/conservation_experiment_plans.tsv (Route B).