Curate & link scRNA-seq datasets#

This illustrates how to manage scRNA-seq datasets in absence of a custom schema.

import lamindb as ln
import lnschema_bionty as lb

# don't recurse through ontology hierarchies to speed up CI
# recommend to set to True
lb.settings.auto_save_parents = False

✅ loaded instance: testuser1/test-scrna (lamindb 0.50.7)

ln.track()

✅ saved: Transform(id='Nv48yAceNSh8z8', name='Curate & link scRNA-seq datasets', short_name='scrna', stem_id='Nv48yAceNSh8', version='0', type=notebook, updated_at=2023-08-17 17:32:36, created_by_id='DzTjkKse')

✅ saved: Run(id='UcskU0EgGleETR6Oao82', run_at=2023-08-17 17:32:36, transform_id='Nv48yAceNSh8z8', created_by_id='DzTjkKse')

Preparation: registries#

Let’s assume that this is not the first time we work with experimental entities, and hence, our registries are already pre-populated:

ln.view(schema="bionty", registries=["CellType", "ExperimentalFactor", "Tissue"])

CellType

	name	ontology_id	abbr	synonyms	description	bionty_source_id	updated_at	created_by_id
id
4fOuOYtl	endothelial cell	CL:0000115	None	endotheliocyte	An Endothelial Cell Comprises The Outermost La...	u0GC	2023-08-17 17:32:40	DzTjkKse
MkrH0gsX	leukocyte	CL:0000738	None	white blood cell\|leucocyte	An Achromatic Cell Of The Myeloid Or Lymphoid ...	u0GC	2023-08-17 17:32:40	DzTjkKse

ExperimentalFactor

	name	ontology_id	abbr	synonyms	description	molecule	instrument	measurement	bionty_source_id	updated_at	created_by_id
id
EfXUa6F0	adult	EFO:0001272	None	adult stage	A Maturity Quality Inhering In An Individual B...	None	None	None	RFxi	2023-08-17 17:32:38	DzTjkKse
eXg039cd	obsolete_C57BL/6	EFO:0004472	None	C57Bl\6\|C57/BL6\|C57BL6\|C57Black\|C57BL/6N\|C57/B...	C57Bl/6 Is A Mouse Strain As Described In Jack...	None	None	None	RFxi	2023-08-17 17:32:37	DzTjkKse

Tissue

	name	ontology_id	abbr	synonyms	description	bionty_source_id	updated_at	created_by_id
id
Na0qNZ1o	inguinal lymph node	UBERON:0001542	None	None	The Lymph Nodes Located In The Groin Area.	ktcN	2023-08-17 17:32:39	DzTjkKse

Mouse lymph node cells: Detmar22#

We’re working with mouse data:

lb.settings.species = "mouse"

✅ set species: Species(id='vado', name='mouse', taxon_id=10090, scientific_name='mus_musculus', updated_at=2023-08-17 17:32:44, bionty_source_id='WiPo', created_by_id='DzTjkKse')

Extract #

Let’s look at a scRNA-seq count matrix in form of an AnnData object:

adata = ln.dev.datasets.anndata_mouse_sc_lymph_node()

Format the metadata columns:

adata

AnnData object with n_obs × n_vars = 1135 × 10000
    obs: 'species', 'ontology_id:species', 'strain', 'ontology_id:strain', 'age', 'ontology_id:age', 'developmental_stage', 'ontology_id:developmental_stage', 'sex', 'ontology_id:sex', 'genotype', 'ontology_id:genotype', 'tissue', 'ontology_id:tissue', 'cell_type', 'ontology_id:cell_type', 'immunophenotype', 'ontology_id:immunophenotype', 'post analysis well quality', 'ontology_id:post analysis well quality', 'Factor Value:single cell identifier', 'Factor Value:Ontology Term[single cell identifier', 'Factor Value:cluster', 'Factor Value:Ontology Term[cluster', 'Factor Value:inferred cell_type - authors labels', 'Factor Value:Ontology Term[inferred cell_type - authors labels'

Curate #

Validate genes in .var:

validated = lb.Gene.validate(adata.var.index, lb.Gene.ensembl_gene_id)

💡 using global setting species = mouse

✅ 9976 terms (99.80%) are validated

❗ 24 terms (0.20%) are not validated

We’re seeing that 24 gene identifiers can’t be validated through Bionty. We’d like to validate all features in this dataset, hence, let’s register them:

non_validated = adata.var.index[~validated]

records = lb.Gene.from_values(non_validated, lb.Gene.ensembl_gene_id)
ln.save(records)

💡 using global setting species = mouse

❗ did not validate 24 Gene records for ensembl_gene_ids: ENSMUSG00000117732, ENSMUSG00000094030, ENSMUSG00000091604, ENSMUSG00000074735, ENSMUSG00000116275, ENSMUSG00000096385, ENSMUSG00000097078, ENSMUSG00000074210, ENSMUSG00000116184, ENSMUSG00000105204, ENSMUSG00000075015, ENSMUSG00000095547, ENSMUSG00000066378, ENSMUSG00000094958, ENSMUSG00000096923, ENSMUSG00000090625, ENSMUSG00000114046, ENSMUSG00000096201, ENSMUSG00000092345, ENSMUSG00000022591, ...

Now all genes pass validation:

lb.Gene.validate(adata.var.index, lb.Gene.ensembl_gene_id);

💡 using global setting species = mouse

✅ 10000 terms (100.00%) are validated

Similarly, for the metadata, we’d like to validate the names as features:

columns = [
    col
    for col in adata.obs.columns
    if not col.startswith("ontology_id") and not col.startswith("Factor Value")
]

ln.save(ln.Feature.from_df(adata.obs[columns]))

❗ did not validate 10 Feature records for names: species, strain, age, developmental_stage, sex, genotype, tissue, cell_type, immunophenotype, post analysis well quality

ln.Feature.validate(columns, field="name");

✅ 10 terms (100.00%) are validated

Some of the metadata can be typed using dedicated registries:

species = lb.Species.from_bionty(name="mouse")
strains = lb.ExperimentalFactor.from_values(adata.obs["strain"], "name")
dev_stages = lb.ExperimentalFactor.from_values(adata.obs["developmental_stage"], "name")
cell_types = lb.CellType.from_values(adata.obs["cell_type"], "name")
tissues = lb.Tissue.from_values(adata.obs["tissue"], "name")

✅ validated 1 Species record on name: mouse

❗ did not validate 1 ExperimentalFactor record for name: C57BL/6N

✅ validated 1 ExperimentalFactor record on name: adult

✅ validated 1 CellType record on name: endothelial cell

✅ validated 1 Tissue record on name: inguinal lymph node

We did not validate strains, hence, let’s try to map synonyms:

lb.ExperimentalFactor.standardize(adata.obs["strain"], return_mapper=True)

{'C57BL/6N': 'obsolete_C57BL/6'}

Indeed, there is a synonym:

adata.obs["strain"] = adata.obs["strain"].map(
    lb.ExperimentalFactor.standardize(adata.obs["strain"], return_mapper=True)
)

Now we can validate:

strains = lb.ExperimentalFactor.from_values(adata.obs["strain"], "name")

✅ validated 1 ExperimentalFactor record on name: obsolete_C57BL/6

Metadata that doesn’t have can’t be typed with dedicated registries:

labels = ln.Label.from_values(adata.obs["sex"])
labels += ln.Label.from_values(adata.obs["age"])
labels += ln.Label.from_values(adata.obs["genotype"])
labels += ln.Label.from_values(adata.obs["immunophenotype"])
ln.save(labels)

❗ did not validate 1 Label record for name: female

❗ did not validate 1 Label record for name: 8 to 10 week

❗ did not validate 1 Label record for name: wild type genotype

❗ did not validate 2 Label records for names: CD45 negative, CD45 positive

Register #

When we create a File object from an AnnData, we’ll automatically link its feature sets and get information about unmapped categories:

file = ln.File.from_anndata(
    adata, description="Detmar22", var_ref=lb.Gene.ensembl_gene_id
)

💡 file will be copied to default storage upon `save()` with key `None` ('.lamindb/9SaWYjVX2DwR0dCC2l1B.h5ad')

💡 parsing feature names of X stored in slot 'var'

💡    using global setting species = mouse

✅    validated 10000 Gene records on ensembl_gene_id: ENSMUSG00000049488, ENSMUSG00000042472, ENSMUSG00000038268, ENSMUSG00000020598, ENSMUSG00000026303, ENSMUSG00000049797, ENSMUSG00000049044, ENSMUSG00000045410, ENSMUSG00000028948, ENSMUSG00000061295, ENSMUSG00000001053, ENSMUSG00000081439, ENSMUSG00000035790, ENSMUSG00000040127, ENSMUSG00000095794, ENSMUSG00000071647, ENSMUSG00000031608, ENSMUSG00000078249, ENSMUSG00000082103, ENSMUSG00000030275, ...

✅    linked: FeatureSet(id='C2hSi0gNZb05Ep0WoIVY', n=10000, type='float', registry='bionty.Gene', hash='T5FFYU5aZpuTyA3bXPfI', created_by_id='DzTjkKse')

💡 parsing feature names of slot 'obs'

✅    validated 10 Feature records on name: age, cell_type, developmental_stage, genotype, immunophenotype, post analysis well quality, sex, species, strain, tissue

❗    did not validate 16 Feature records for names: Factor Value:Ontology Term[cluster, Factor Value:Ontology Term[inferred cell_type - authors labels, Factor Value:Ontology Term[single cell identifier, Factor Value:cluster, Factor Value:inferred cell_type - authors labels, Factor Value:single cell identifier, ontology_id:age, ontology_id:cell_type, ontology_id:developmental_stage, ontology_id:genotype, ontology_id:immunophenotype, ontology_id:post analysis well quality, ontology_id:sex, ontology_id:species, ontology_id:strain, ontology_id:tissue

❗    ignoring non-validated features: Factor Value:Ontology Term[cluster,Factor Value:Ontology Term[inferred cell_type - authors labels,Factor Value:Ontology Term[single cell identifier,Factor Value:cluster,Factor Value:inferred cell_type - authors labels,Factor Value:single cell identifier,ontology_id:age,ontology_id:cell_type,ontology_id:developmental_stage,ontology_id:genotype,ontology_id:immunophenotype,ontology_id:post analysis well quality,ontology_id:sex,ontology_id:species,ontology_id:strain,ontology_id:tissue

✅    linked: FeatureSet(id='pNw2lrfCKq8CRfiBC8oa', n=10, registry='core.Feature', hash='xp0klzB1xrIvoFUodvli', created_by_id='DzTjkKse')

file.save()

✅ saved 2 feature sets for slots: ['var', 'obs']

✅ storing file '9SaWYjVX2DwR0dCC2l1B' at '.lamindb/9SaWYjVX2DwR0dCC2l1B.h5ad'

The file now has two linked feature sets:

file.features

'var': FeatureSet(id='C2hSi0gNZb05Ep0WoIVY', n=10000, type='float', registry='bionty.Gene', hash='T5FFYU5aZpuTyA3bXPfI', updated_at=2023-08-17 17:32:53, created_by_id='DzTjkKse')
'obs': FeatureSet(id='pNw2lrfCKq8CRfiBC8oa', n=10, registry='core.Feature', hash='xp0klzB1xrIvoFUodvli', updated_at=2023-08-17 17:32:54, created_by_id='DzTjkKse')

file.add_labels(species, feature="species")
file.add_labels(strains + dev_stages + tissues + cell_types)

✅ linked feature 'species' to registry 'bionty.Species'

✅ linked labels 'obsolete_C57BL/6' to feature 'strain', linked feature 'strain' to registry 'bionty.ExperimentalFactor'

✅ linked labels 'adult' to feature 'developmental_stage', linked feature 'developmental_stage' to registry 'bionty.ExperimentalFactor'

✅ linked labels 'inguinal lymph node' to feature 'tissue', linked feature 'tissue' to registry 'bionty.Tissue'

✅ linked labels 'endothelial cell' to feature 'cell_type', linked feature 'cell_type' to registry 'bionty.CellType'

file.add_labels(labels)

✅ linked labels 'female' to feature 'sex', linked feature 'sex' to registry 'core.Label'

✅ linked labels '8 to 10 week' to feature 'age', linked feature 'age' to registry 'core.Label'

✅ linked labels 'wild type genotype' to feature 'genotype', linked feature 'genotype' to registry 'core.Label'

✅ linked labels 'CD45 negative', 'CD45 positive' to feature 'immunophenotype', linked feature 'immunophenotype' to registry 'core.Label'

The file is now queryable by everything we linked:

file.describe()

💡 File(id=9SaWYjVX2DwR0dCC2l1B, key=None, suffix=.h5ad, accessor=AnnData, description=Detmar22, version=None, size=17342743, hash=lC0PrTQici8k6yWBFsaJzg, hash_type=md5, created_at=2023-08-17 17:32:54.663207+00:00, updated_at=2023-08-17 17:32:54.663230+00:00)

Provenance:
    🗃️ storage: Storage(id='6ObKC1Wo', root='/home/runner/work/lamin-usecases/lamin-usecases/docs/test-scrna', type='local', updated_at=2023-08-17 17:32:30, created_by_id='DzTjkKse')
    📔 transform: Transform(id='Nv48yAceNSh8z8', name='Curate & link scRNA-seq datasets', short_name='scrna', stem_id='Nv48yAceNSh8', version='0', type='notebook', updated_at=2023-08-17 17:32:53, created_by_id='DzTjkKse')
    👣 run: Run(id='UcskU0EgGleETR6Oao82', run_at=2023-08-17 17:32:36, transform_id='Nv48yAceNSh8z8', created_by_id='DzTjkKse')
    👤 created_by: User(id='DzTjkKse', handle='testuser1', email='testuser1@lamin.ai', name='Test User1', updated_at=2023-08-17 17:32:30)
Features:
  var (X):
    🔗 index (10000, bionty.Gene.id): ['M1FVY5laHAth', 'B0mmBd1BIj73', 'WqZ5QZ4HkbO8', 'wFKdeJV97lNv', 'QrWPtoLL0HTX'...]
  obs (metadata):
    🔗 cell_type (1, bionty.CellType): ['endothelial cell']
    🔗 strain (2, bionty.ExperimentalFactor): ['obsolete_C57BL/6', 'adult']
    🔗 developmental_stage (2, bionty.ExperimentalFactor): ['obsolete_C57BL/6', 'adult']
    🔗 species (1, bionty.Species): ['mouse']
    🔗 tissue (1, bionty.Tissue): ['inguinal lymph node']
    🔗 genotype (1, core.Label): ['wild type genotype']
    🔗 age (1, core.Label): ['8 to 10 week']
    🔗 sex (1, core.Label): ['female']
    🔗 immunophenotype (2, core.Label): ['CD45 positive', 'CD45 negative']

Human immune cells: Conde22#

lb.settings.species = "human"

conde22 = ln.dev.datasets.anndata_human_immune_cells()

ln.save(lb.Gene.from_values(conde22.var.index, lb.Gene.ensembl_gene_id))

💡 using global setting species = human

✅ validated 36390 Gene records from Bionty on ensembl_gene_id: ENSG00000243485, ENSG00000237613, ENSG00000186092, ENSG00000238009, ENSG00000239945, ENSG00000239906, ENSG00000241860, ENSG00000241599, ENSG00000286448, ENSG00000236601, ENSG00000284733, ENSG00000235146, ENSG00000284662, ENSG00000229905, ENSG00000237491, ENSG00000177757, ENSG00000228794, ENSG00000225880, ENSG00000230368, ENSG00000272438, ...

❗ did not validate 113 Gene records for ensembl_gene_ids: ENSG00000112096, ENSG00000182230, ENSG00000203812, ENSG00000204092, ENSG00000215271, ENSG00000221995, ENSG00000224739, ENSG00000224745, ENSG00000225932, ENSG00000226377, ENSG00000226380, ENSG00000226403, ENSG00000227021, ENSG00000227220, ENSG00000227902, ENSG00000228139, ENSG00000228906, ENSG00000229352, ENSG00000231575, ENSG00000232196, ...

conde22.obs.columns = conde22.obs.columns.str.replace("donor_id", "donor")
columns = [col for col in conde22.obs.columns if "ontology_term" not in col]
ln.save(ln.Feature.from_df(conde22.obs[columns]))

✅ validated 2 Feature records on name: cell_type, tissue

❗ did not validate 2 Feature records for names: assay, donor

file = ln.File.from_anndata(
    conde22, description="Conde22", var_ref=lb.Gene.ensembl_gene_id
)

💡 file will be copied to default storage upon `save()` with key `None` ('.lamindb/P8fI3RY9UtTAEQfY7wx1.h5ad')

💡 parsing feature names of X stored in slot 'var'

💡    using global setting species = human

✅    validated 36503 Gene records on ensembl_gene_id: ENSG00000115539, ENSG00000224359, ENSG00000253399, ENSG00000283247, ENSG00000083828, ENSG00000231505, ENSG00000267278, ENSG00000100028, ENSG00000187833, ENSG00000284710, ENSG00000105672, ENSG00000267670, ENSG00000126787, ENSG00000125352, ENSG00000212122, ENSG00000185730, ENSG00000254526, ENSG00000231265, ENSG00000287355, ENSG00000251055, ...

✅    linked: FeatureSet(id='zUz2mA7XBIB20eZr9Oiq', n=36503, type='float', registry='bionty.Gene', hash='dnRexHCtxtmOU81_EpoJ', created_by_id='DzTjkKse')

💡 parsing feature names of slot 'obs'

✅    validated 4 Feature records on name: assay, cell_type, donor, tissue

❗    did not validate 3 Feature records for names: assay_ontology_term_id, cell_type_ontology_term_id, tissue_ontology_term_id

❗    ignoring non-validated features: assay_ontology_term_id,cell_type_ontology_term_id,tissue_ontology_term_id

✅    linked: FeatureSet(id='n6WJW6D4Kf8VxORkqwrR', n=4, registry='core.Feature', hash='dVn-3OuJyOEjmXLMT-cd', created_by_id='DzTjkKse')

file.save()

✅ saved 2 feature sets for slots: ['var', 'obs']

✅ storing file 'P8fI3RY9UtTAEQfY7wx1' at '.lamindb/P8fI3RY9UtTAEQfY7wx1.h5ad'

The file has the following linked feature sets:

file.features

'var': FeatureSet(id='zUz2mA7XBIB20eZr9Oiq', n=36503, type='float', registry='bionty.Gene', hash='dnRexHCtxtmOU81_EpoJ', updated_at=2023-08-17 17:33:08, created_by_id='DzTjkKse')
'obs': FeatureSet(id='n6WJW6D4Kf8VxORkqwrR', n=4, registry='core.Feature', hash='dVn-3OuJyOEjmXLMT-cd', updated_at=2023-08-17 17:33:12, created_by_id='DzTjkKse')

Let’s now link observational metadata.

cell_types = lb.CellType.from_values(conde22.obs.cell_type, field="name")
efs = lb.ExperimentalFactor.from_values(conde22.obs.assay, field="name")
tissues = lb.Tissue.from_values(conde22.obs.tissue, field="name")

✅ validated 32 CellType records from Bionty on name: classical monocyte, T follicular helper cell, memory B cell, alveolar macrophage, naive thymus-derived CD4-positive, alpha-beta T cell, effector memory CD8-positive, alpha-beta T cell, terminally differentiated, alpha-beta T cell, CD4-positive helper T cell, naive thymus-derived CD8-positive, alpha-beta T cell, macrophage, mucosal invariant T cell, group 3 innate lymphoid cell, naive B cell, animal cell, CD16-negative, CD56-bright natural killer cell, human, plasma cell, CD8-positive, alpha-beta memory T cell, CD16-positive, CD56-dim natural killer cell, human, gamma-delta T cell, conventional dendritic cell, ...

✅ validated 3 ExperimentalFactor records from Bionty on name: 10x 3' v3, 10x 5' v2, 10x 5' v1

✅ validated 17 Tissue records from Bionty on name: blood, thoracic lymph node, spleen, lung, mesenteric lymph node, lamina propria, liver, jejunal epithelium, omentum, bone marrow, ileum, caecum, thymus, skeletal muscle tissue, duodenum, sigmoid colon, transverse colon

file.add_labels([lb.settings.species], feature="species")
ln.save(cell_types + efs + tissues)
file.add_labels(cell_types + efs + tissues)

✅ created feature set for slot 'external'

✅ linked labels 'classical monocyte', 'T follicular helper cell', 'memory B cell', 'alveolar macrophage', 'naive thymus-derived CD4-positive, alpha-beta T cell', 'effector memory CD8-positive, alpha-beta T cell, terminally differentiated', 'alpha-beta T cell', 'CD4-positive helper T cell', 'naive thymus-derived CD8-positive, alpha-beta T cell', 'macrophage', 'mucosal invariant T cell', 'group 3 innate lymphoid cell', 'naive B cell', 'animal cell', 'CD16-negative, CD56-bright natural killer cell, human', 'plasma cell', 'CD8-positive, alpha-beta memory T cell', 'CD16-positive, CD56-dim natural killer cell, human', 'gamma-delta T cell', 'conventional dendritic cell', 'CD8-positive, alpha-beta memory T cell, CD45RO-positive', 'effector memory CD4-positive, alpha-beta T cell', 'non-classical monocyte', 'mast cell', 'regulatory T cell', 'progenitor cell', 'dendritic cell, human', 'plasmablast', 'plasmacytoid dendritic cell', 'lymphocyte', 'germinal center B cell', 'megakaryocyte' to feature 'cell_type'

✅ linked labels '10x 3' v3', '10x 5' v2', '10x 5' v1' to feature 'assay', linked feature 'assay' to registry 'bionty.ExperimentalFactor'

✅ linked labels 'blood', 'thoracic lymph node', 'spleen', 'lung', 'mesenteric lymph node', 'lamina propria', 'liver', 'jejunal epithelium', 'omentum', 'bone marrow', 'ileum', 'caecum', 'thymus', 'skeletal muscle tissue', 'duodenum', 'sigmoid colon', 'transverse colon' to feature 'tissue'

As neither the core schema nor lnschema_bionty have a Donor table, we’re using Label to track donor ids:

donors = ln.Label.from_values(conde22.obs["donor"])
ln.save(donors)
file.add_labels(donors)

❗ did not validate 12 Label records for names: D496, 621B, A29, A36, A35, 637C, A52, A37, D503, 640C, A31, 582C

✅ linked labels 'D496', '621B', 'A29', 'A36', 'A35', '637C', 'A52', 'A37', 'D503', '640C', 'A31', '582C' to feature 'donor', linked feature 'donor' to registry 'core.Label'

file.describe()

💡 File(id=P8fI3RY9UtTAEQfY7wx1, key=None, suffix=.h5ad, accessor=AnnData, description=Conde22, version=None, size=28061905, hash=3cIcmoqp1MxjX8NlRkKGlQ, hash_type=md5, created_at=2023-08-17 17:33:12.867793+00:00, updated_at=2023-08-17 17:33:12.867819+00:00)

Provenance:
    🗃️ storage: Storage(id='6ObKC1Wo', root='/home/runner/work/lamin-usecases/lamin-usecases/docs/test-scrna', type='local', updated_at=2023-08-17 17:32:30, created_by_id='DzTjkKse')
    📔 transform: Transform(id='Nv48yAceNSh8z8', name='Curate & link scRNA-seq datasets', short_name='scrna', stem_id='Nv48yAceNSh8', version='0', type='notebook', updated_at=2023-08-17 17:33:08, created_by_id='DzTjkKse')
    👣 run: Run(id='UcskU0EgGleETR6Oao82', run_at=2023-08-17 17:32:36, transform_id='Nv48yAceNSh8z8', created_by_id='DzTjkKse')
    👤 created_by: User(id='DzTjkKse', handle='testuser1', email='testuser1@lamin.ai', name='Test User1', updated_at=2023-08-17 17:32:30)
Features:
  var (X):
    🔗 index (36503, bionty.Gene.id): ['a1AGqGk4ywu1', 'dNgfeMyI0d1p', 'br4ZjTdWVCNF', '3Ucw70JttwA8', '9bqHaU58Tjf1'...]
  external:
    🔗 species (1, bionty.Species): ['human']
  obs (metadata):
    🔗 cell_type (32, bionty.CellType): ['macrophage', 'T follicular helper cell', 'plasma cell', 'regulatory T cell', 'animal cell']
    🔗 assay (3, bionty.ExperimentalFactor): ["10x 5' v2", "10x 5' v1", "10x 3' v3"]
    🔗 tissue (17, bionty.Tissue): ['liver', 'lamina propria', 'mesenteric lymph node', 'omentum', 'spleen']
    🔗 donor (12, core.Label): ['A37', 'A31', 'A52', '637C', 'A35']

A less well curated dataset#

Let’s now consider a dataset with less-well curated features:

pbcm68k = ln.dev.datasets.anndata_pbmc68k_reduced()

We see that this dataset is indexed by gene symbols:

pbcm68k.var.index

Index(['HES4', 'TNFRSF4', 'SSU72', 'PARK7', 'RBP7', 'SRM', 'MAD2L2', 'AGTRAP',
       'TNFRSF1B', 'EFHD2',
       ...
       'ATP5O', 'MRPS6', 'TTC3', 'U2AF1', 'CSTB', 'SUMO3', 'ITGB2', 'S100B',
       'PRMT2', 'MT-ND3'],
      dtype='object', name='index', length=765)

Because gene symbols don’t uniquely characterize an Ensembl ID, we’re linking more feature records to this file than columns in the AnnData.

Tip

Use Ensembl Gene IDs rather than gene Symbols to index genes.

validated = lb.Gene.validate(pbcm68k.var.index, lb.Gene.symbol)

💡 using global setting species = human

✅ 695 terms (90.80%) are validated

❗ 70 terms (9.20%) are not validated

pbcm68k_validated = pbcm68k[:, validated]

Link cell types:

Now, all cell types should be validated:

lb.CellType.validate(pbcm68k_validated.obs["cell_type"], "name");

❗ received 9 unique terms, 61 empty/duplicated terms are ignored

✅ 9 terms (100.00%) are validated

file_pbcm68k = ln.File.from_anndata(
    pbcm68k_validated, description="10x reference pbmc68k", var_ref=lb.Gene.symbol
)

💡 file will be copied to default storage upon `save()` with key `None` ('.lamindb/oVfV4tXeKpGM6X7UFMk7.h5ad')

💡 parsing feature names of X stored in slot 'var'

💡    using global setting species = human

✅    validated 695 Gene records on symbol: PPP1R14A, RSL1D1, KLF6, CD47, BBX, OTUB1, GATA3, DCXR, SRRM2, IL2RB, BANK1, G0S2, ATP6V1E1, IMPDH2, AGTRAP, SLCO3A1, HLA-DPB1, ARPC5L, ACADS, ATP6V1G1, ...

✅    linked: FeatureSet(id='fgqRxj5dmLeN5groV2zW', n=695, type='float', registry='bionty.Gene', hash='W4ps_86b5dxk2Wd1gWTo', created_by_id='DzTjkKse')

💡 parsing feature names of slot 'obs'

✅    validated 1 Feature record on name: cell_type

❗    did not validate 3 Feature records for names: louvain, n_genes, percent_mito

❗    ignoring non-validated features: louvain,n_genes,percent_mito

✅    linked: FeatureSet(id='gGvGY3rlFIqEN7LXUlAG', n=1, registry='core.Feature', hash='Fhzyzfg9Cd5MnYB_uSjG', created_by_id='DzTjkKse')

file_pbcm68k.save()

✅ saved 2 feature sets for slots: ['var', 'obs']

✅ storing file 'oVfV4tXeKpGM6X7UFMk7' at '.lamindb/oVfV4tXeKpGM6X7UFMk7.h5ad'

cell_types = lb.CellType.from_values(pbcm68k_validated.obs["cell_type"], "name")
file_pbcm68k.add_labels(cell_types)

✅ validated 9 CellType records on name: B cell, CD19-positive, CD14-positive, CD16-negative classical monocyte, CD16-positive, CD56-dim natural killer cell, human, CD38-negative naive B cell, CD8-positive, CD25-positive, alpha-beta regulatory T cell, conventional dendritic cell, cytotoxic T cell, dendritic cell, effector memory CD4-positive, alpha-beta T cell, terminally differentiated

✅ linked labels 'B cell, CD19-positive', 'CD14-positive, CD16-negative classical monocyte', 'CD16-positive, CD56-dim natural killer cell, human', 'CD38-negative naive B cell', 'CD8-positive, CD25-positive, alpha-beta regulatory T cell', 'conventional dendritic cell', 'cytotoxic T cell', 'dendritic cell', 'effector memory CD4-positive, alpha-beta T cell, terminally differentiated' to feature 'cell_type'

file_pbcm68k.describe()

💡 File(id=oVfV4tXeKpGM6X7UFMk7, key=None, suffix=.h5ad, accessor=AnnData, description=10x reference pbmc68k, version=None, size=589484, hash=eKVXV5okt5YRYjySMTKGEw, hash_type=md5, created_at=2023-08-17 17:33:21.798553+00:00, updated_at=2023-08-17 17:33:21.798576+00:00)

Provenance:
    🗃️ storage: Storage(id='6ObKC1Wo', root='/home/runner/work/lamin-usecases/lamin-usecases/docs/test-scrna', type='local', updated_at=2023-08-17 17:32:30, created_by_id='DzTjkKse')
    📔 transform: Transform(id='Nv48yAceNSh8z8', name='Curate & link scRNA-seq datasets', short_name='scrna', stem_id='Nv48yAceNSh8', version='0', type='notebook', updated_at=2023-08-17 17:33:21, created_by_id='DzTjkKse')
    👣 run: Run(id='UcskU0EgGleETR6Oao82', run_at=2023-08-17 17:32:36, transform_id='Nv48yAceNSh8z8', created_by_id='DzTjkKse')
    👤 created_by: User(id='DzTjkKse', handle='testuser1', email='testuser1@lamin.ai', name='Test User1', updated_at=2023-08-17 17:32:30)
Features:
  var (X):
    🔗 index (695, bionty.Gene.id): ['S40ynPlt1FR6', 'NlRk1RxzXPJS', '10ogBGiTBuv4', 'E231nvehewhz', 'Z2SBxpiWYpPs'...]
  obs (metadata):
    🔗 cell_type (9, bionty.CellType): ['CD8-positive, CD25-positive, alpha-beta regulatory T cell', 'cytotoxic T cell', 'effector memory CD4-positive, alpha-beta T cell, terminally differentiated', 'CD14-positive, CD16-negative classical monocyte', 'CD38-negative naive B cell']

file_pbcm68k.view_lineage()

https://d33wubrfki0l68.cloudfront.net/bc4b6176dcefa356db44620829538448919a810b/8ce96/_images/b24b62163f38771d3dce867e5c6eb3447ce4e35ee4a69dabd2cf56f4afc3801f.svg

🎉 Now let’s continue with data integration: Integrate scRNA-seq datasets