Curate & link flow cytometry data

Show code cell content Hide code cell content

!lamin init --storage ./test-flow --schema bionty

💡 creating schemas: core==0.45.5 bionty==0.29.6 

✅ saved: User(id='DzTjkKse', handle='testuser1', email='testuser1@lamin.ai', name='Test User1', updated_at=2023-08-17 17:34:06)
✅ saved: Storage(id='zcQDVCAz', root='/home/runner/work/lamin-usecases/lamin-usecases/docs/test-flow', type='local', updated_at=2023-08-17 17:34:06, created_by_id='DzTjkKse')

✅ loaded instance: testuser1/test-flow
💡 did not register local instance on hub (if you want, call `lamin register`)

import lamindb as ln
import lnschema_bionty as lb
import readfcs

lb.settings.species = "human"  # globally set species

Show code cell output Hide code cell output

✅ loaded instance: testuser1/test-flow (lamindb 0.50.7)

✅ set species: Species(id='uHJU', name='human', taxon_id=9606, scientific_name='homo_sapiens', updated_at=2023-08-17 17:34:08, bionty_source_id='bLIp', created_by_id='DzTjkKse')

ln.track()

💡 notebook imports: lamindb==0.50.7 lnschema_bionty==0.29.6 readfcs==1.1.5

✅ saved: Transform(id='OWuTtS4SAponz8', name='Curate & link flow cytometry data', short_name='flow', stem_id='OWuTtS4SApon', version='0', type=notebook, updated_at=2023-08-17 17:34:08, created_by_id='DzTjkKse')

✅ saved: Run(id='UCZcXSzMoEOP50mA88rF', run_at=2023-08-17 17:34:08, transform_id='OWuTtS4SAponz8', created_by_id='DzTjkKse')

We start with a flow cytometry file from Alpert19:

ln.dev.datasets.file_fcs_alpert19()

PosixPath('Alpert19.fcs')

Use readfcs to read the fcs file into memory:

adata = readfcs.read("Alpert19.fcs")
adata

AnnData object with n_obs × n_vars = 166537 × 40
    var: 'n', 'channel', 'marker', '$PnB', '$PnE', '$PnR'
    uns: 'meta'

Track data with cell markers

We’ll use the CellMarker reference to link features:

file = ln.File.from_anndata(adata, description="Alpert19", var_ref=lb.CellMarker.name)

💡 file will be copied to default storage upon `save()` with key `None` ('.lamindb/8IDiVJdryJNs8opm1rnF.h5ad')

💡 parsing feature names of X stored in slot 'var'

💡    using global setting species = human

✅    validated 27 CellMarker records from Bionty on name: CD57, CD8, CD85j, CD11c, CD16, CD3, CD38, CD27, CD94, CD86, CXCR5, CXCR3, CD45RA, CD20, CD127, CD33, CD28, CD24, ICOS, CD161, ...

❗    did not validate 13 CellMarker records for names: (Ba138)Dd, Bead, CCR6, CCR7, CD11b, CD14, CD19, CD4, Cell_length, Dead, IgD, PD-1, Time

❗    ignoring non-validated features: (Ba138)Dd,Bead,CCR6,CCR7,CD11b,CD14,CD19,CD4,Cell_length,Dead,IgD,PD-1,Time

✅    linked: FeatureSet(id='Pu8TaDqztwQdL0kTux9k', n=27, type='float', registry='bionty.CellMarker', hash='B4S1GzQjtfW_90rnrQ71', created_by_id='DzTjkKse')

We see that many features aren’t validated. Let’s standardize the identifiers:

adata.var.index = lb.CellMarker.standardize(adata.var.index)

💡 using global setting species = human

❗ found 8 synonyms in Bionty: ['CD19', 'CD4', 'IgD', 'CD11b', 'CD14', 'CCR6', 'CCR7', 'PD-1']
   please add corresponding records via `.from_values(['Cd19', 'Cd4', 'Igd', 'CD11B', 'Cd14', 'Ccr6', 'Ccr7', 'PD1'])`

Now things look much better, but we still have 5 CellMaker records that seem more like metadata.

file = ln.File.from_anndata(adata, description="Alpert19", var_ref=lb.CellMarker.name)

Show code cell output Hide code cell output

💡 file will be copied to default storage upon `save()` with key `None` ('.lamindb/fo6BymjbSaGvq9sPIZf5.h5ad')

💡 parsing feature names of X stored in slot 'var'

💡    using global setting species = human

✅    validated 35 CellMarker records from Bionty on name: CD57, Cd19, Cd4, CD8, Igd, CD85j, CD11c, CD16, CD3, CD38, CD27, CD11B, Cd14, Ccr6, CD94, CD86, CXCR5, CXCR3, Ccr7, CD45RA, ...

❗    did not validate 5 CellMarker records for names: (Ba138)Dd, Bead, Cell_length, Dead, Time

❗    ignoring non-validated features: (Ba138)Dd,Bead,Cell_length,Dead,Time

✅    linked: FeatureSet(id='ZRa4RjU6DrVhUddolLal', n=35, type='float', registry='bionty.CellMarker', hash='ldY9_GmptHLCcT7Nrpgo', created_by_id='DzTjkKse')

Hence, let’s curate the AnnData a bit more:

validated = lb.CellMarker.bionty().validate(adata.var.index, "name")

💡 using global setting species = human

✅ 35 terms (87.50%) are validated

❗ 5 terms (12.50%) are not validated

Let’s move metadata (non-validated cell markers) into adata.obs:

adata.obs = adata[:, ~validated].to_df()
adata = adata[:, validated].copy()

Now we have a clean panel of 35 CellMarkers and metadata that we don’t want to register:

file = ln.File.from_anndata(adata, description="Alpert19", var_ref=lb.CellMarker.name)

💡 file will be copied to default storage upon `save()` with key `None` ('.lamindb/1ZX3ZAuXF660c14ebDjW.h5ad')

💡 parsing feature names of X stored in slot 'var'

💡    using global setting species = human

✅    validated 35 CellMarker records from Bionty on name: CD57, Cd19, Cd4, CD8, Igd, CD85j, CD11c, CD16, CD3, CD38, CD27, CD11B, Cd14, Ccr6, CD94, CD86, CXCR5, CXCR3, Ccr7, CD45RA, ...

✅    linked: FeatureSet(id='9eNgudvrOkWU6xR4NQL2', n=35, type='float', registry='bionty.CellMarker', hash='ldY9_GmptHLCcT7Nrpgo', created_by_id='DzTjkKse')

💡 parsing feature names of slot 'obs'

❗    did not validate 5 Feature records for names: Time, Cell_length, Dead, (Ba138)Dd, Bead

❗    ignoring non-validated features: Time,Cell_length,Dead,(Ba138)Dd,Bead

❗    no validated features, skip creating feature set

file.save()

✅ saved 1 feature set for slot: ['var']

✅ storing file '1ZX3ZAuXF660c14ebDjW' at '.lamindb/1ZX3ZAuXF660c14ebDjW.h5ad'

file.features

'var': FeatureSet(id='9eNgudvrOkWU6xR4NQL2', n=35, type='float', registry='bionty.CellMarker', hash='ldY9_GmptHLCcT7Nrpgo', updated_at=2023-08-17 17:34:16, created_by_id='DzTjkKse')

file.features["var"].df().head(10)

	name	synonyms	gene_symbol	ncbi_gene_id	uniprotkb_id	species_id	bionty_source_id	updated_at	created_by_id
id
50v4SaR2m5zQ	CD25		IL2RA	3559	P01589	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
cFJEI6e6wml3	CD20		MS4A1	931	A0A024R507	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
CLFUvJpioHoA	CD28		CD28	940	B4E0L1	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
fpPkjlGv15C9	Ccr6		CCR6	1235	P51684	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
0qCmUijBeByY	CD94		KLRD1	3824	Q13241	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
sYcK7uoWCtco	Ccr7		CCR7	1236	P32248	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
ljp5UfCF9HCi	TCRgd	TCRGAMMADELTA|TCRγδ	None	None	None	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
hVNEgxlcDV10	CD127		IL7R	3575	P16871	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
0evamYEdmaoY	Igd		None	None	None	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
L0m6f7FPiDeg	CD86		CD86	942	A8K632	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse

Let’s register another flow file:

adata2 = readfcs.read(ln.dev.datasets.file_fcs())
file2 = ln.File.from_anndata(
    adata2, description="My fcs file", var_ref=lb.CellMarker.name
)

💡 file will be copied to default storage upon `save()` with key `None` ('.lamindb/EPYEa463ci0UrpFhpyKc.h5ad')

💡 parsing feature names of X stored in slot 'var'

💡    using global setting species = human

✅    validated 6 CellMarker records on name: CD57, CD8, CD3, CD27, CD127, CD28

✅    validated 3 CellMarker records from Bionty on name: CCR5, CD45RO, SSC-A

❗    did not validate 7 CellMarker records for names: CCR7, CD14, CD19, CD4, FSC-A, FSC-H, KI67

❗    ignoring non-validated features: CCR7,CD14,CD19,CD4,FSC-A,FSC-H,KI67

✅    linked: FeatureSet(id='ZknXPHbPqKhuPYZSWPUH', n=9, type='float', registry='bionty.CellMarker', hash='i1PcDEtXiGpkDIwfdMp3', created_by_id='DzTjkKse')

adata2.var.index = lb.CellMarker.standardize(adata2.var.index)

💡 using global setting species = human

❗ found 1 synonym in Bionty: ['KI67']
   please add corresponding records via `.from_values(['Ki67'])`

file2 = ln.File.from_anndata(
    adata2, description="My fcs file", var_ref=lb.CellMarker.name
)

💡 file will be copied to default storage upon `save()` with key `None` ('.lamindb/7mQLeOy9I8qnwXouSHdl.h5ad')

💡 parsing feature names of X stored in slot 'var'

💡    using global setting species = human

✅    validated 10 CellMarker records on name: CD57, Cd19, Cd4, CD8, CD3, CD27, Cd14, Ccr7, CD127, CD28

✅    validated 4 CellMarker records from Bionty on name: CCR5, CD45RO, Ki67, SSC-A

❗    did not validate 2 CellMarker records for names: FSC-A, FSC-H

❗    ignoring non-validated features: FSC-A,FSC-H

✅    linked: FeatureSet(id='0YuydfGcWl3rqHmCf8r3', n=14, type='float', registry='bionty.CellMarker', hash='npy5P7AYbjKLInpXlNvb', created_by_id='DzTjkKse')

file2.save()

✅ saved 1 feature set for slot: ['var']

✅ storing file '7mQLeOy9I8qnwXouSHdl' at '.lamindb/7mQLeOy9I8qnwXouSHdl.h5ad'

file2.view_lineage()

Query by cell markers

Which datasets have CD14 in the flow panel:

cell_markers = lb.CellMarker.lookup()

cell_markers.cd14

CellMarker(id='roEbL8zuLC5k', name='Cd14', synonyms='', gene_symbol='CD14', ncbi_gene_id='4695', uniprotkb_id='O43678', updated_at=2023-08-17 17:34:16, species_id='uHJU', bionty_source_id='MhSj', created_by_id='DzTjkKse')

panels_with_cd14 = ln.FeatureSet.filter(cell_markers=cell_markers.cd14).all()

ln.File.filter(feature_sets__in=panels_with_cd14).df()

	storage_id	key	suffix	accessor	description	version	initial_version_id	size	hash	hash_type	transform_id	run_id	updated_at	created_by_id
id
7mQLeOy9I8qnwXouSHdl	zcQDVCAz	None	.h5ad	AnnData	My fcs file	None	None	6876232	Cf4Fhfw_RDMtKd5amM6Gtw	md5	OWuTtS4SAponz8	UCZcXSzMoEOP50mA88rF	2023-08-17 17:34:20	DzTjkKse
1ZX3ZAuXF660c14ebDjW	zcQDVCAz	None	.h5ad	AnnData	Alpert19	None	None	33371192	mmo80O4vZUTVpBC064VIDw	md5	OWuTtS4SAponz8	UCZcXSzMoEOP50mA88rF	2023-08-17 17:34:16	DzTjkKse

Shared cell markers between two files:

files = ln.File.filter(feature_sets__in=panels_with_cd14).list()
file1, file2 = files[0], files[1]

file1_markers = file1.features["var"]
file2_markers = file2.features["var"]

shared_markers = file1_markers & file2_markers
shared_markers.list("name")

['CD3', 'CD28', 'Cd4', 'CD127', 'CD27', 'Cd19', 'Cd14', 'CD8', 'CD57', 'Ccr7']

Flow marker registry

Check out your CellMarker registry:

lb.CellMarker.filter().df()

	name	synonyms	gene_symbol	ncbi_gene_id	uniprotkb_id	species_id	bionty_source_id	updated_at	created_by_id
id
Nb2sscq9cBcB	CD57		B3GAT1	27087	Q9P2W7	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
8OhpfB7wwV32	Cd19		CD19	930	P15391	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
HEK41hvaIazP	Cd4		CD4	920	B4DT49	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
ttBc0Fs01sYk	CD8		CD8A	925	P01732	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
0evamYEdmaoY	Igd		None	None	None	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
lRZYuH929QDw	CD85j		None	None	None	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
L0WKZ3fufq0J	CD11c		ITGAX	3687	P20702	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
bspnQ0igku6c	CD16		FCGR3A	2215	O75015	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
a4hvNp34IYP0	CD3		None	None	None	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
CR7DAHxybgyi	CD38		CD38	952	B4E006	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
uThe3c0V3d4i	CD27		CD27	939	P26842	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
N2F6Qv9CxJch	CD11B		ITGAM	3684	P11215	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
roEbL8zuLC5k	Cd14		CD14	4695	O43678	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
fpPkjlGv15C9	Ccr6		CCR6	1235	P51684	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
0qCmUijBeByY	CD94		KLRD1	3824	Q13241	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
L0m6f7FPiDeg	CD86		CD86	942	A8K632	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
4uiPHmCPV5i1	CXCR5		CXCR5	643	A0N0R2	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
agQD0dEzuoNA	CXCR3		CXCR3	2833	P49682	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
sYcK7uoWCtco	Ccr7		CCR7	1236	P32248	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
a624IeIqbchl	CD45RA		None	None	None	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
cFJEI6e6wml3	CD20		MS4A1	931	A0A024R507	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
hVNEgxlcDV10	CD127		IL7R	3575	P16871	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
c3dZKHFOdllB	CD33		CD33	945	P20138	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
CLFUvJpioHoA	CD28		CD28	940	B4E0L1	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
gEfe8qTsIHl0	CD24		CD24	100133941	B6EC88	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
0vAls2cmLKWq	ICOS		ICOS	29851	Q53QY6	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
4EojtgN0CjBH	CD161		KLRB1	3820	Q12918	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
ljp5UfCF9HCi	TCRgd	TCRGAMMADELTA|TCRγδ	None	None	None	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
2VeZenLi2dj5	PD1	PID1|PD-1|PD 1	PDCD1	5133	A0A0M3M0G7	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
n40112OuX7Cq	CD123		IL3RA	3563	P26951	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
h4rkCALR5WfU	CD56		NCAM1	4684	P13591	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
k0zGbSgZEX3q	HLADR	HLA‐DR|HLA-DR|HLA DR	None	None	None	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
50v4SaR2m5zQ	CD25		IL2RA	3559	P01589	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
YA5Ezh6SAy10	DNA1		None	None	None	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
yCyTIVxZkIUz	DNA2		DNA2	1763	P51530	uHJU	MhSj	2023-08-17 17:34:16	DzTjkKse
UMsp5g0fgMwY	CCR5		CCR5	1234	P51681	uHJU	MhSj	2023-08-17 17:34:20	DzTjkKse
XvpJ6oL3SG7w	CD45RO		None	None	None	uHJU	MhSj	2023-08-17 17:34:20	DzTjkKse
Qa4ozz9tyesQ	Ki67	Ki-67|KI 67	None	None	None	uHJU	MhSj	2023-08-17 17:34:20	DzTjkKse
VZBURNy04vBi	SSC-A	SSC A|SSCA	None	None	None	uHJU	MhSj	2023-08-17 17:34:20	DzTjkKse

Show code cell content Hide code cell content

# a few tests
assert set(shared_markers.list("name")) == set(
    [
        "Ccr7",
        "CD3",
        "Cd14",
        "Cd19",
        "CD127",
        "CD27",
        "CD28",
        "CD8",
        "Cd4",
        "CD57",
    ]
)
ln.File.filter(feature_sets__in=panels_with_cd14).exists()

True

Show code cell content Hide code cell content

# clean up test instance
!lamin delete --force test-flow
!rm -r test-flow

💡 deleting instance testuser1/test-flow
✅     deleted instance settings file: /home/runner/.lamin/instance--testuser1--test-flow.env

✅     instance cache deleted
✅     deleted '.lndb' sqlite file
❗     consider manually deleting your stored data: /home/runner/work/lamin-usecases/lamin-usecases/docs/test-flow