scTypeEval :microscope: :chart_with_upwards_trend:

A Framework for Evaluating Single-Cell Transcriptomics Cell Type Annotations

Accurate cell type annotation is essential but difficult in single-cell RNA sequencing (scRNA-seq).

• Manual approaches are time-consuming, subjective, and inconsistent.
• Automated classifiers are faster but often fail to generalize across tissues, conditions, or closely related cell types.
• A key limitation for both is the lack of true ground truth for benchmarking.

scTypeEval provides a ground-truth-agnostic framework to systematically assess annotation quality using internal validation metrics.

• Quantifies inter-sample label consistency (ISC).
• Identifies ambiguous or misclassified populations.
• Enables reproducible benchmarking of manual annotations, automated classifiers, or clustering results.

:key: Key Features

• Ground-truth agnostic – Evaluate annotations without external references.
• Cross-dataset benchmarking – Assess consistency across samples and studies.
• Customizable – Works with Seurat, SCE, or matrices; supports custom gene lists and parameters.
• Robust – Sensitive to misclassification; reliable across batch effects, label granularity, and sample sizes.

Documentation & Resources

For comprehensive tutorials, detailed metric explanations, and real-world case studies, please visit our documentation website:
scTypeEval Tutorials and Case Studies

Installation

Install scTypeEval from Bioconductor (recommended) or GitHub.

# Option 1 (recommended): Bioconductor/devel install (R >= 4.6.0)
if (!requireNamespace("BiocManager", quietly = TRUE)) {
  install.packages("BiocManager")
}
BiocManager::install("scTypeEval")

# Option 2: Install GitHub releases
if (!requireNamespace("remotes", quietly = TRUE)) {
  install.packages("remotes")
}
# Installing without @<tag> pulls the current GitHub branch (R >= 4.6.0)
# remotes::install_github("carmonalab/scTypeEval")
install.packages("scTypeEval", repos = c('https://carmonalab.r-universe.dev', 'https://cloud.r-project.org'))

# R-4.3.0_prerelease installs the pre-release compatible with R >= 4.3.0
remotes::install_github("carmonalab/scTypeEval@R-4.3.0_prerelease")

Typical install time on desktop computer

• Only scTypeEval package: ~10 seconds.
• First-time Bioconductor install (including dependencies): typically ~5-10 minutes.

Usage

To run scTypeEval on your own data, follow the workflow below.

For complete runnable examples, see the repository vignettes:

• Full workflow vignette: vignettes/scTypeEval.Rmd - Runtime: 16s
• Quick start vignette: vignettes/quick-start.Rmd - Runtime: 2s

For larger datasets, runtime increases with dataset size (especially number of samples), feature space, and selected dissimilarity method:

• Pseudobulk-based methods are typically very fast (often milliseconds to seconds).
• Reciprocal classification methods from seconds to a few minutes.
• Wasserstein-based methods are usually slower and can require several minutes.

library(scTypeEval)

# From count matrix and metadata dataframe
sceval <- create_scTypeEval(matrix = count_matrix,
                           metadata = metadata)

# From Seurat object
sceval <- create_scTypeEval(seurat_obj)

# From SCE object
sceval <- create_scTypeEval(sce)

# Run data processing on an scTypeEval object
sceval <- run_processing_data(
  scTypeEval = sceval,
  ident = "celltype",        # column in metadata defining identities (e.g. cell type)
  sample = "patient_id",     # column in metadata defining sample IDs
  min_samples = 5,           # minimum samples required to retain a cell type
  min_cells = 10,            # minimum cells required per sample-celltype
)

# Identify highly variable genes (HVGs)
sceval <- run_hvg(
  scTypeEval = sceval,
  var_method = "scran",   # method to compute HVGs
  ngenes = 2000,          # number of HVGs to retain
  sample = TRUE   # whether to perform sample-level blocking
)

# Identify marker genes per cell type
sceval <- run_gene_markers(
  scTypeEval = sceval,
  method = "scran.findMarkers", # supported: scran.findMarkers
  ngenes_celltype = 50          # max markers per cell type
)

sceval <- run_pca(
  scTypeEval = sceval,
  ndim = 30          # number of PCs
)

# Euclidean distance based on pseudobulk aggregation
sceval <- run_dissimilarity(sceval,
                       method = "Pseudobulk:Euclidean",
                       reduction = FALSE) # whether to compute dissimilarities in low dimensional space

# Wasserstein distance on embeddings
sceval <- run_dissimilarity(sceval,
                       method = "WasserStein",
                       reduction = TRUE)

# Reciprocal Classification Match using SingleR classifier
sceval <- run_dissimilarity(sceval,
                       method = "recip_classif:Match",
                       reciprocal_classifier = "SingleR")

plot_heatmap(sceval,
            dissimilarity_slot = "recip_classif:Match",
            sort_consistency = "silhouette",
            sort_similarity = "Pseudobulk:Euclidean")

consis <- get_consistency(scTypeEval,
                          dissimilarity_slot = "Pseudobulk:Euclidean", # indicate in which dissimilarity space compute metrics
                          consistency_metric = "silhouette" # choose consistency metric
                          )

Recommended metrics

Based on our benchmark across 31 real scRNA-seq datasets, we recommend the following ISC metric choices:

By default (and in the benchmark), these metrics are computed on highly variable genes (HVGs) identified with scran via run_hvg(var_method = "scran").

In our benchmark we observed a trade-off: some metrics are more sensitive to global, gradual signal degradation, while others are more sensitive to local structural inconsistencies/over-partitioning, so we recommend using one from each class.

• Local consistency (over-partitioning / fragmentation sensitive): silhouette in the reciprocal-classification match space (recip_classif:Match) — also the top-ranked overall choice in the benchmark.
• Global consistency (signal degradation sensitive): 2label_silhouette in pseudobulk cosine space (Pseudobulk:Cosine).
• Integrated score: combine local + global ISC score by taking the geometric mean. In some datasets with high granularity (e.g. only CD8+ T cells subsets) usage of only Local consistency is recommended.

Minimal code to compute the recommended metrics on your own dataset:

library(dplyr)
library(scTypeEval)

## Assumes you already created `sceval` and ran processing (see Usage above)

# Default feature space used in the benchmark: scran HVGs, sample-blocked,
# on single-cell data
# Also, we recommend removing from selected genes within HVG the black listed
# genes with high inter-sample variability
data(black_list)
bl <- list(black_list$TCR,
           black_list$Immunoglobulins,
           black_list$Ygenes) |>
       unlist()
       
sceval <- run_hvg(
  scTypeEval = sceval,
  var_method = "scran",
  ngenes = 2000,
  sample = TRUE,
  aggregation = "single-cell",
  black_list = bl
)

# Not affecting recip_classif:Match but for Pseudobulk-based metrics,
# computation in PCA space is faster with similar results
# (See Usage above to add dimensional reduction).

# (1) Overall best and local structure: silhouette in
# reciprocal-classification match space
sceval <- run_dissimilarity(
  sceval,
  method = "recip_classif:Match",
  reduction = FALSE
)
# (2) Complementary for global structure: 2-label silhouette 
# in pseudobulk cosine space
sceval <- run_dissimilarity(
  sceval,
  method = "Pseudobulk:Cosine",
  reduction = TRUE
)
# (3) Obtain consistency scores for computed dissimilarities and
# for silhouette and 2label_silhouette
isc <- get_consistency(
  sceval,
  dissimilarity_slot = c("recip_classif:Match", "Pseudobulk:Cosine"),
  consistency_metric = c("silhouette", "2label_silhouette")
)

# (4) Filter to obtain best combinations for each cell type
sel.methods <- c("local" = "silhouette | recip_classif:Match",
                "global" = "2label_silhouette | Pseudobulk:Cosine")

isc |>
  mutate(method.type = paste(consistency_metric,
                             dissimilarity_method,
                             sep = " | ")
                             ) |>
  filter(method.type %in% sel.methods)

Reproduction instructions

To reproduce the manuscript benchmarking analyses and figures (task simulations + scoring helpers), use:

• ISC_benchmark_reproducibility

This external repository contains the dedicated code and instructions for full quantitative reproduction.

Citation

If you use scTypeEval in your work, please cite the associated preprint:

Garnica J, Andreatta M, Carmona SJ (2026). Evaluating cell type annotations in single-cell omics in the absence of ground truth. bioRxiv. https://doi.org/10.64898/2026.06.10.731285

This work is currently available as a preprint; a peer-reviewed publication will be added when available.