Datasets#
univi.datasets downloads the datasets used in the tutorials and the Genome Research article, verifies each file against its registered MD5 checksum, and caches it in ~/.cache/univi/<name>/ (or under UNIVI_DATA_DIR). Every dataset is a Zenodo record of .h5ad files, one per modality or cohort.
import univi.datasets as uds
uds.list_datasets() # one row per dataset: files, cells, size, figures
data = uds.pbmc_multiome_10k() # {"rna": AnnData, "atac": AnnData}
data = uds.load("pbmc_multiome_10k", keys=["rna"])
uds.dataset_info("pbmc_multiome_10k") # registry entry: description, DOI, files, checksums
Please cite the studies that generated the data as well as UniVI (full references on the citation page).
Catalog#
Dataset |
Assay |
Keys |
Cells |
Download |
Paper |
Zenodo |
|---|---|---|---|---|---|---|
CITE-seq, human PBMCs |
|
161,764 |
799 MB |
Figs. 2–3; S1 |
||
10x Multiome, human PBMCs |
|
9,631 |
205 MB |
Figs. 4, 8–10; S2, S8–S11; Supp. Notebook S1 |
||
10x Multiome reference + scRNA-seq + scATAC-seq, human PBMCs |
|
12,012 paired; 30,495 RNA; 47,148 ATAC |
699 MB |
Fig. 5; S5 |
||
TEA-seq, human PBMCs |
|
29,894 |
420 MB |
Fig. 6; S6 |
||
CITE-seq + scRNA-seq + DAb-seq, human AML |
|
43,179 paired; 37,627 RNA; 40,239 protein |
300 MB |
Fig. 7; S7 |
||
SHARE-seq, mouse skin |
|
31,220 |
293 MB |
S3 |
||
scNMT-seq, mouse embryos |
|
1,140 |
224 MB |
S4 |
Download sizes are the sum of the registered file sizes.
What the files contain#
The hosted files are the objects the archived analysis notebooks worked with, taken after quality control, cell filtering, and annotation, and before any learned transform (feature selection, normalization, scaling, TF-IDF/LSI). Fit those transforms on the training cells yourself; the API notebooks show how for each analysis.
.Xholds raw counts, with two exceptions: the scNMT-seq CpG and GpC files hold methylation and accessibility fractions (with the read counts in layers), and the DAb-seq file ofaml_mosaicholds the processed protein values that came with the original DAb-seq objects.Paired keys contain the same cells, and
loadreturns them in identical order. Keys that are independent cohorts (for exampleding_rnainpbmc_multiome_bridge) are returned as stored.layers["counts"]: if.Xholds integer counts and there is nocountslayer,loadcopies.Xthere, which is where the UniVI preprocessors look. Passcounts_layer=Noneto leave the objects untouched.obs["split"]is present when the analysis used a fixed train/validation/test assignment. Otherwise the dataset’s section says how the split is built.
Each section below lists the files, what .X and .obs hold, how the split is defined, and the original data. The collapsed How these files were exported block gives the export recipe: run the named archived notebook (in notebooks/GR_manuscript_reproducibility/) up to the cell given, then run the snippet in a new cell. Every snippet starts with:
from univi.datasets import export_dataset
hao_citeseq_pbmc#
Paired RNA and surface-protein (CITE-seq) profiles of 161,764 human PBMCs from Hao et al. (2021), with the authors’ three-level cell-type annotation. Used for Figs. 2–3 and Supplemental Fig. S1.
Record: Zenodo 10.5281/zenodo.22883697 · Original data: Hao et al. (2021), GEO GSE164378 · Notebook: Paired CITE-seq, CITE-seq tutorial
data = uds.hao_citeseq_pbmc() # same as uds.load("hao_citeseq_pbmc")
rna, adt = data["rna"], data["adt"]
Key |
File |
Cells |
Features |
|
|
|---|---|---|---|---|---|
|
hao_citeseq_pbmc_rna.h5ad (748.7 MB) |
161,764 |
20,729 genes |
Raw counts |
|
|
hao_citeseq_pbmc_adt.h5ad (49.9 MB) |
161,764 |
228 antibody-derived tags |
Raw ADT counts |
Same columns as |
Split. obs["split"] holds the split used in the article: stratified by celltype.l3, at most 1,200 cells per type in the train/validation pool (80/10), and all remaining cells in test, which gives 37,862 / 4,722 / 119,180 cells.
How these files were exported
Notebook UniVI_manuscript_GR-Figure__3__CITE_paired_biological_latent.ipynb, through the cell that defines train_idx, val_idx, test_idx:
export_dataset(
"hao_citeseq_pbmc", {"rna": rna, "adt": adt}, "zenodo/hao_citeseq_pbmc",
labels=["celltype.l1", "celltype.l2", "celltype.l3"],
splits={"train": train_idx, "val": val_idx, "test": test_idx},
title="Hao et al. (2021) PBMC CITE-seq (RNA + ADT)", source="GEO GSE164378",
)
pbmc_multiome_10k#
Paired RNA and chromatin accessibility (10x Multiome) for 9,631 human PBMCs after QC, annotated in obs["cell_type"] by Seurat label transfer against the Hao et al. (2021) PBMC reference. Used for Figs. 4 and 8–10, Supplemental Figs. S2 and S8–S11, and Supplemental Notebook S1, and by the quickstart, query mapping, supervised heads, and generation tutorials.
Record: Zenodo 10.5281/zenodo.19581816 (CC BY 4.0) · Original data: 10x Genomics, “PBMC from a Healthy Donor – Granulocytes Removed Through Cell Sorting (10k)”, Cell Ranger ARC 1.0.0 · Notebook: Paired 10x Multiome
data = uds.pbmc_multiome_10k() # same as uds.load("pbmc_multiome_10k")
rna, atac = data["rna"], data["atac"]
Key |
File |
Cells |
Features |
|
|
|---|---|---|---|---|---|
|
10x-Multiome-Pbmc10k-RNA.h5ad (50.7 MB) |
9,631 |
29,095 genes |
Raw counts |
|
|
10x-Multiome-Pbmc10k-ATAC.h5ad (153.9 MB) |
9,631 |
107,194 peaks ( |
Raw peak counts |
Same columns as |
Split. None is stored. The Fig. 4 analysis stratifies by cell_type, capping each type at 800 training and 100 validation cells (80/10 otherwise) and sending the rest to test: split_by_label(rna.obs["cell_type"], train_fraction=0.8, val_fraction=0.1, train_cap=800, val_cap=100, seed=0) gives 5,777 / 717 / 3,137 cells. The quickstart uses an uncapped stratified 80/10/10 split.
These two files were published before export_dataset existed and are the same files the archived Fig. 4 notebook reads; the Zenodo record describes their provenance.
pbmc_multiome_bridge#
Three human PBMC cohorts for the bridge analysis: a paired 10x Multiome reference with no curated labels (12,012 cells, keys rna and atac), an independent scRNA-seq cohort from Ding et al. (2020) (30,495 cells), and an independent scATAC-seq cohort from Satpathy et al. (2019) (47,148 cells). The two cohorts carry harmonized cell-type labels, which the analysis uses for supervised refinement and then transfers back to the reference. Used for Fig. 5 and Supplemental Fig. S5.
Record: Zenodo 10.5281/zenodo.22883390 · Original data: 10x Genomics (Multiome reference); Ding et al. (2020), GEO GSE132044; Satpathy et al. (2019), GEO GSE129785 · Notebook: Bridging unpaired RNA and ATAC cohorts
data = uds.load("pbmc_multiome_bridge")
rna, atac = data["rna"], data["atac"] # paired reference
ding, satpathy = data["ding_rna"], data["satpathy_atac"] # unpaired query cohorts
Key |
File |
Cells |
Features |
|
|
|---|---|---|---|---|---|
|
pbmc_multiome_bridge_rna.h5ad (46.6 MB) |
12,012 |
9,022 genes |
Raw counts |
|
|
pbmc_multiome_bridge_atac.h5ad (113.6 MB) |
12,012 |
85,485 peaks |
Raw peak counts |
Same columns as |
|
pbmc_multiome_bridge_ding_rna.h5ad (66.4 MB) |
30,495 |
94,507 (see below) |
Raw counts |
|
|
pbmc_multiome_bridge_satpathy_atac.h5ad (472.3 MB) |
47,148 |
94,507 (see below) |
Raw counts |
|
The two query files share one feature space of 94,507 features, the size of the reference’s genes and peaks combined (9,022 + 85,485): ding_rna contains all 9,022 reference genes and satpathy_atac all 85,485 reference peaks. Select each cohort’s own features before preprocessing, as the Fig. 5 notebook does:
ding = ding[:, rna.var_names].copy()
satpathy = satpathy[:, atac.var_names].copy()
Split. obs["split"] on the reference holds the archived notebook’s random 90/10 train/validation assignment (seed 42): 10,810 training and 1,201 validation cells, plus one leftover cell marked unassigned. The Ding and Satpathy cohorts are the query sets and have no split.
How these files were exported
Notebook UniVI_manuscript_GR-Figure__5__Multiome_bridge_mapping_and_fine-tuning.ipynb, through the cell that adds celltype_harmonized_coarse. The Multiome reference has no curated cell-type labels; the classifier head is trained on the harmonized labels of the Ding and Satpathy cohorts, so confirm those columns first:
objs = {"rna": rna, "atac": atac, "ding_rna": uni_rna, "satpathy_atac": uni_atac}
for k, a in objs.items():
print(k, a.shape, [c for c in a.obs.columns if "celltype" in c])
export_dataset(
"pbmc_multiome_bridge", objs, "zenodo/pbmc_multiome_bridge", paired=["rna", "atac"],
labels=["celltype_harmonized_coarse"], splits={"train": idx_train, "val": idx_val},
title="10x Multiome PBMC reference with Ding et al. (2020) scRNA-seq and Satpathy et al. (2019) scATAC-seq",
source="10x Genomics; GEO GSE132044; GEO GSE129785",
)
teaseq_pbmc#
Trimodal TEA-seq profiles of human PBMCs from Swanson et al. (2021): RNA, surface protein, and chromatin accessibility measured in the same 29,894 cells across four wells. Used for Fig. 6 and Supplemental Fig. S6.
Record: Zenodo 10.5281/zenodo.22883520 · Original data: Swanson et al. (2021), GEO GSE158013 · Notebook: Trimodal TEA-seq, held-out well
data = uds.load("teaseq_pbmc")
rna, adt, atac = data["rna"], data["adt"], data["atac"]
Key |
File |
Cells |
Features |
|
|
|---|---|---|---|---|---|
|
teaseq_pbmc_rna.h5ad (76.5 MB) |
29,894 |
36,601 genes |
Raw counts |
|
|
teaseq_pbmc_adt.h5ad (4.7 MB) |
29,894 |
47 antibody-derived tags (those measured in all four wells) |
Raw ADT counts |
|
|
teaseq_pbmc_atac.h5ad (339.0 MB) |
29,894 |
155,930 genome-wide 500-bp tiles (open in 0.5–80% of training-well cells) |
Raw tile counts |
|
The three files store the same cells in the same order.
Split. None is stored; the split is by well. The article trains on wells 3, 4, and 6 and evaluates on held-out well 5 (sample_id == "GSM5123953_X066-MP0C1W5_leukopak_perm-cells_tea"). The API notebook holds out a random 10% of training-well cells for early stopping, giving 20,226 training, 2,247 validation, and 7,421 held-out cells.
How these files were exported
Notebook UniVI_manuscript_GR-Figure__6__TEA-seq_tri-modal.ipynb, through the ATAC cell that builds the tile matrices (tile_mats). The per-well raw objects are stacked, the protein panel is restricted to markers present in every well, and tiles are filtered as in the notebook (open in 0.5–80% of training-well cells), which keeps the file a manageable size:
import anndata as ad
import numpy as np
wells = list(teaseq_prefixes)
def stack(parts):
out = ad.concat(parts, label="_well", keys=wells, index_unique=None)
out.obs["sample_id"] = out.obs.pop("_well").astype(str)
return out
rna_parts = [rna_raw_dict[p].copy() for p in wells]
for a in rna_parts:
a.var_names_make_unique()
panel = sorted(set.intersection(*[set(adt_raw_dict[p].var_names) for p in wells]))
rna_all = stack(rna_parts)
adt_all = stack([adt_raw_dict[p][:, panel].copy() for p in wells])
atac_all = stack([ad.AnnData(X=tile_mats[p][1], obs=tile_mats[p][0].obs.copy(), var=tile_mats[p][0].var.copy())
for p in wells])
in_train = (atac_all.obs["sample_id"] != HOLDOUT_PREFIX).to_numpy()
frac = np.asarray((atac_all.X[in_train] > 0).mean(axis=0)).ravel()
atac_all = atac_all[:, (frac >= MIN_TILE_FRAC) & (frac <= MAX_TILE_FRAC)]
common = rna_all.obs_names.intersection(adt_all.obs_names).intersection(atac_all.obs_names)
export_dataset(
"teaseq_pbmc", {"rna": rna_all[common], "adt": adt_all[common], "atac": atac_all[common]}, "zenodo/teaseq_pbmc",
title="TEA-seq PBMCs (RNA + ADT + ATAC tiles), four wells", source="GEO GSE158013",
)
aml_mosaic#
A mosaic of three acute myeloid leukemia cohorts that were never measured together: paired CITE-seq from Knorr et al. (2023) (43,179 cells from 11 bone-marrow samples, 8 newly diagnosed AML and 3 age-matched normal donors; keys cite_rna and cite_adt), which trains the RNA–protein bridge; scRNA-seq from van Galen et al. (2019) (37,627 cells) with per-cell mutant and wild-type transcript calls; and DAb-seq from Demaree et al. (2021) (40,239 cells) with surface protein and single-cell genotype calls. Used for Fig. 7 and Supplemental Fig. S7.
Record: Zenodo 10.5281/zenodo.22883601 · Original data: Knorr et al. (2023), GEO GSE220473 (the CITE-seq SubSeries of SuperSeries GSE220474); van Galen et al. (2019), GEO GSE116256; Demaree et al. (2021), BioProject PRJNA602320 · Notebook: AML mosaic integration and mutation heads
data = uds.load("aml_mosaic")
cite_rna, cite_adt = data["cite_rna"], data["cite_adt"] # paired bridge
vg, dab = data["vangalen_rna"], data["dabseq_adt"] # RNA-only and protein-only cohorts
Key |
File |
Cells |
Features |
|
|
|---|---|---|---|---|---|
|
aml_mosaic_cite_rna.h5ad (190.7 MB) |
43,179 |
21,782 genes (the same genes as |
Raw counts |
|
|
aml_mosaic_cite_adt.h5ad (3.8 MB) |
43,179 |
18 surface markers (the same panel as |
Raw ADT counts |
Same columns as |
|
aml_mosaic_vangalen_rna.h5ad (97.1 MB) |
37,627 |
21,782 genes |
Raw counts |
|
|
aml_mosaic_dabseq_adt.h5ad (8.4 MB) |
40,239 |
18 surface markers |
Processed protein values from the original objects (not counts) |
Genotype calls |
Genotype truth differs by cohort. For van Galen cells it is derived per gene from the MutTranscripts/WtTranscripts strings; DAb-seq cells have direct calls for NPM1, DNMT3A, and FLT3 only, stored in the variant-level columns of the original objects (there are no gene-level mut_<GENE> columns), so other genes should be treated as unmeasured for them rather than wild type. Because dabseq_adt does not hold counts, load adds no counts layer to it; the Fig. 7 notebook standardizes those values within the cohort.
Split. obs["split"] in cite_rna and cite_adt assigns whole samples: 8 training, 1 validation, and 2 test samples (28,414 / 4,652 / 10,113 cells). vangalen_rna and dabseq_adt have no split; the API notebook’s mutation heads hold out whole van Galen patients (orig.ident) and DAb-seq experiments (experiment).
How these files were exported
Notebook UniVI_manuscript_GR-Figure__7__AML_bridge_mapping_and_fine-tuning.ipynb. Right after the cell that defines the grouped splits, keep a copy of them:
aml_splits = {k: list(splits[k]) for k in ("train_obs", "val_obs", "test_obs")}
Then export. dab_adt_al keeps the processed protein values and the variant-level genotype columns (NPM1 W288fs, DNMT3A R882H, FLT3-ITD) of the original DAb-seq objects, as in the hosted file; van Galen cells keep their MutTranscripts / WtTranscripts fields:
export_dataset(
"aml_mosaic",
{"cite_rna": cite_rna_al, "cite_adt": cite_adt_al, "vangalen_rna": vg_rna_al, "dabseq_adt": dab_adt_al},
"zenodo/aml_mosaic", paired=["cite_rna", "cite_adt"], labels=["sample_id"],
splits={"train": aml_splits["train_obs"], "val": aml_splits["val_obs"], "test": aml_splits["test_obs"]},
title="AML mosaic: CITE-seq bridge (Knorr et al. 2023), van Galen et al. (2019) scRNA-seq, DAb-seq (Demaree et al. 2021)",
source="GEO GSE220473; GEO GSE116256; BioProject PRJNA602320",
)
scnmt_gastrulation#
Paired RNA, CpG methylation, and GpC accessibility (scNMT-seq) for 1,140 cells from mouse gastrulation (Argelaguet et al. 2019), with CpG and GpC signal summarized over gene bodies. Used for Supplemental Fig. S4.
Record: Zenodo 10.5281/zenodo.22885463 (files, manifest, and release_audit.tsv) · Original data: Argelaguet et al. (2019), GEO GSE121708 · Notebook: scNMT-seq mouse gastrulation
from univi.datasets import build_univi_inputs_from_scnmt_triplet
data = uds.scnmt_gastrulation() # same as uds.load("scnmt_gastrulation")
rna, cpg, gpc = data["rna"], data["cpg"], data["gpc"]
built = build_univi_inputs_from_scnmt_triplet(rna, cpg, gpc)
adata_dict = built["adata_dict"]
recon_targets_spec = built["recon_targets_spec"]
Key |
File |
Cells |
Features |
|
Layers |
|---|---|---|---|---|---|
|
scnmt_gastrulation_rna.h5ad (46.8 MB) |
1,140 |
22,084 genes |
Raw counts |
|
|
scnmt_gastrulation_cpg.h5ad (74.4 MB) |
1,140 |
18,285 gene bodies |
Methylation fractions |
|
|
scnmt_gastrulation_gpc.h5ad (102.8 MB) |
1,140 |
18,325 gene bodies |
Accessibility fractions |
|
The three files contain the same cells in the same order, and each carries the EBI sample metadata (stage with 4 values, lineage10x with 23, lineage10x_2, embryo, plate, and the pass_rnaQC / pass_metQC / pass_accQC flags) plus split; the CpG and GpC files add per-cell coverage_total and n_features_covered. The success and coverage layers are what the beta-binomial likelihoods use: build_univi_inputs_from_scnmt_triplet log-normalizes RNA while keeping the raw counts, and returns the reconstruction targets that point the CpG and GpC decoders at those layers. The 33 GB source archive is not needed for this workflow.
Split. obs["split"] holds the article’s seed-0 85/5/10 split (969 / 57 / 114 cells).
How these files were built
Unlike the other datasets, these files were not written with export_dataset, which keeps only .X. They were parsed from the EBI’s feature-level bundle (Argelaguet et al. 2019) with the reader settings of the archived Supplemental Fig. S4 notebook (no QC or feature filtering, minimum coverage 1 for CpG and GpC) and written directly, keeping the layers the beta-binomial likelihoods need:
from univi.datasets import load_scnmt_gastrulation_genebody_triplet
triplet = load_scnmt_gastrulation_genebody_triplet("scnmt_gastrulation.tar.gz", require_qc=False,
filter_features=False, min_cov_cpg=1, min_cov_gpc=1)
# obs["split"]: seed-0 85/5/10 split; then write each object with .write_h5ad(..., compression="gzip")
Loading options#
uds.load(name, keys=None, *, data_dir=None, counts_layer="counts", force=False, progress=True) downloads what is missing, verifies it, and returns {key: AnnData}. keys loads a subset of files, data_dir overrides UNIVI_DATA_DIR, and force=True downloads again. A cached file is checked against its MD5 each time it is used and downloaded again if it does not match. uds.fetch(name, keys) does the download and verification only and returns {key: Path}, which is convenient for pointing other tools at the cached files.
Using your own or mirrored data#
Register files for the current session, or point UNIVI_DATASET_REGISTRY at a JSON file with the same structure (useful for lab mirrors or offline clusters):
uds.register_dataset("my_cite", {
"paired": ["rna", "adt"],
"files": {
"rna": {"filename": "rna.h5ad", "url": "https://example.org/rna.h5ad", "hash": "sha256:..."},
"adt": {"filename": "adt.h5ad", "url": "https://example.org/adt.h5ad", "hash": "sha256:..."},
},
})
data = uds.load("my_cite")
url may be https://, http://, ftp:// or file://; urls accepts a list of mirrors tried in order. Hashes are written md5:<hex> or sha256:<hex>.
Packaging and publishing a dataset#
export_dataset is how the hosted datasets above were written. It stores raw counts in .X, the selected .obs columns, and an optional split in obs["split"], one .h5ad per key, together with a manifest.json (checksums, shapes, split counts) and a DESCRIPTION.md for the record:
uds.export_dataset("my_cite", {"rna": rna, "adt": adt}, "export/my_cite",
labels=["cell_type"], splits={"train": train_ids, "val": val_ids, "test": test_ids})
scripts/zenodo_upload.py uploads the exported files to a Zenodo draft, and scripts/prepare_zenodo_release.py register adds the published record to the registry:
python scripts/zenodo_upload.py zenodo/hao_citeseq_pbmc --creator "Ashford, Andrew J." --orcid 0000-0002-1234-2118
# review the draft on zenodo.org, then publish it there
python scripts/prepare_zenodo_release.py register zenodo/hao_citeseq_pbmc --record-id <record id>
python -c "import univi.datasets as u; print(u.list_datasets())"
zenodo_upload.py needs a Zenodo token with the deposit:write scope in the ZENODO_TOKEN environment variable; --sandbox does a dry run on sandbox.zenodo.org. It verifies each uploaded file’s checksum and links the record to the article’s DOI. register writes the download URLs and MD5 checksums into univi/datasets/registry.json, which ships with the next UniVI release. Until then, the updated registry file can be used directly with the UNIVI_DATASET_REGISTRY environment variable.
Before publishing, confirm that each original source permits redistribution of processed data, and keep the source citations in the record description.