31:T5c3,The `sdk.clustjs` module is based on the ClustJS SDK: https://github.com/lorenasandoval88/clustjs General data utilities, clustering, PCA, heatmaps, and plotting. --- ### Why this SDK is needed - D3.js is excellent for visualization, but it does **not** include built-in PCA or clustering algorithms. - Observable Plot is useful for plotting, but it does **not** compute PCA or hierarchical clustering by itself. - Plotly.js can display interactive plots, but the core JavaScript library does **not** provide a built-in PCA engine like some Python/R workflows. - Therefore, browser-based multivariate analysis requires combining visualization libraries with math/statistics libraries. `ClustJS` fills this gap by providing browser-native utilities for: - computing PCA - running hierarchical clustering - generating heatmaps - preparing numeric data matrices - visualizing results directly in JavaScript --- ### Key features - Browser-native **data analysis and visualization** - Supports: - PCA (dimensionality reduction) - hierarchical clustering - heatmaps and pair plots - Built on **D3.js** for interactive graphics - Designed for **genomic and multivariate data workflows** --- ### Built-in data ``` sdk.clustjs.irisData sdk.clustjs.pcaDt sdk.clustjs.pairsDt sdk.clustjs.hclustDt ``` ### data analysis functions ``` sdk.clustjs.pca_plot(options) sdk.clustjs.hclust_plot(options) sdk.clustjs.heatmap_plot(options) sdk.clustjs.pairs_plot(options) ``` ---32:Tac5,The `sdk.pgp` module is based on the Personal Genomes Project SDK: https://github.com/lorenasandoval88/personal_genomes_project_sdk It provides browser-side helper functions for discovering public PGP data types, fetching participant metadata, loading profile records, and downloading publicly available datasets, including 23andMe-style genotype files, biometric data, health records, images, microbiome data, and other miscellaneous files. While the Personal Genome Project (PGP) supports multiple public data categories, this SDK currently focuses on retrieving and parsing 23andMe-style genotype files to enable browser-based polygenic risk score (PRS) construction, clustering, and downstream exploratory analysis. ### Why this SDK is needed - The Personal Genomes Project (PGP) does **not provide a stable, fully documented REST API** - Public data is exposed through: - HTML pages (not structured APIs) - semi-structured JSON endpoints - redirect-based download links - Download links often: - **redirect (302)** to external storage (e.g., Arvados) - may change over time - Direct browser access is difficult due to: - CORS restrictions - inconsistent response formats - pagination and scraping challenges As a result, working with PGP data in JavaScript requires custom logic for: - discovering available datasets - parsing metadata from multiple formats - resolving redirected download URLs - standardizing outputs for analysis --- ### How `sdk.pgp` fills this gap: - standardizes PGP data access - simplifies participant and file discovery - abstracts away redirects and parsing complexity - enables direct loading of 23andMe genotype files - integrates seamlessly into PRS and clustering workflows --- ### Key features - Browser-native access to **public PGP genomic datasets** - Supports discovery of **23andMe-style genotype files** - Designed for **client-side workflows** (no backend required) - Integrates with ClustJS pipelines for PRS + clustering --- ### Caching summary - Uses **`localforage` (IndexedDB)** for browser-side caching - Reduces repeated network requests to PGP endpoints - Improves performance in Observable notebooks and demos - Typical cached objects include: - participant metadata (`pgp:23andme-allUsers`) - available data types (`pgp:dataTypes`) - profile data (`pgp:profiles:`) - Cache helps mitigate: - slow page scraping - unstable endpoints - repeated redirects for downloads --- The following provide a **structured wrapper layer** for working with PGP data in the browser. ### Functions ``` sdk.pgp.fetch23andMeParticipants(limit=10) sdk.pgp.allUsersMetaDataByType_fast("dataType") sdk.pgp.fetchAvailableDataTypes() sdk.pgp.fetchProfile(["ids"]) ``` ---33:T9e6,The `sdk.pgs` module is based on the PGS Catalog SDK: https://github.com/lorenasandoval88/pgs_catalog_sdk It provides functions for loading PGS Catalog score metadata, traits, scoring files, and cache/storage utilities. ### Why this SDK is needed - The PGS Catalog provides multiple endpoints (scores, traits, scoring files), but: - linking scores ↔ traits requires **manual joins across endpoints** - fetching full mappings often requires **many API calls per score** - The full score list (`fetchAllScores`) can contain: - duplicate entries - inconsistent or typo-prone metadata - Trait data (`fetchTraits`) is more stable and structured, but: - does not directly provide complete score metadata - Data access is further complicated by: - reliance on **PGS Catalog REST endpoints** - large responses (~5,000 scores, ~600+ traits) - pagination requirements - evolving API structure (counts, fields, categories) - Scoring files introduce additional challenges: - large file sizes (`.txt` / `.txt.gz`) - require parsing in the browser - repeated downloads are expensive and slow Naively building a **trait → PGS → scoring file pipeline** requires: - thousands of API calls - repeated large file downloads - significant latency in browser environments --- ### How `sdk.pgs` fills this gap - **Combines full score metadata with trait data** - builds **trait → PGS mappings in a single pass** - avoids calling each score ID individually - **Standardizes and cleans metadata** - reduces impact of duplicates and inconsistencies - **Caches both metadata and scoring files** - uses `localforage` (IndexedDB) - avoids repeated downloads of large `.txt` scoring files - enables fast reloads - **Optimizes browser-based workflows** - reduces network overhead - minimizes API calls - supports scalable PRS computation in the browser ### Key features - Browser-native access to **PGS Catalog data** - Supports loading: - score metadata (~5,000 PGS) - trait metadata (~600+ traits) - scoring files (variant-level effect weights) - Enables **PRS construction workflows** - Integrates with ClustJS for downstream clustering and visualization --- The following provide a **structured wrapper layer** for working with PGP data in the browser. ### Functions ``` sdk.pgs.getTxts(["ids"]) //"PGS000001" sdk.pgs..fetchAllScores() // ~5,000 sdk.pgs.fetchTraits() // ~669 sdk.pgs.getScoresPerTrait() //~11 sdk.pgs.getScoresPerCategory() sdk.pgs.fetchSomeScores(["ids"]) ```34:T1ad1,The sdk.prs module is based on the PRS SDK: https://github.com/lorenasandoval88/polygenic_risk_scores It provides browser-native functions for matching PGS scoring files to 23andMe genotype data and computing Polygenic Risk Scores (PRS), including variant matching, allele counting, and quality control. ### Why this SDK is needed - PGS scoring files and 23andMe genotype files are **not directly compatible**: - PGS files use standardized fields (`rsID`, `hm_chr`, `hm_pos`, `effect_allele`, `effect_weight`) - 23andMe files use a different structure (`chromosome`, `position`, `genotype`) - Variant matching requires: - resolving **rsID vs chr:position** - handling **strand ambiguity (A/T, C/G)** - filtering valid genotypes and alleles - Many variants in a PGS model will: - not exist in a given 23andMe file - require explicit tracking for quality control - PRS computation is not just a simple sum: - requires **allele counting (0,1,2)** - multiplication by **effect weights** - aggregation across matched variants - careful handling of missing data - Real-world PGS data introduces additional challenges: - inconsistent column naming across files - missing or malformed values - large scoring files (hundreds–thousands of SNPs) - need for reproducible QC checks Implementing PRS in the browser requires: - custom parsing logic for both file types - complex variant matching pipelines - repeated debugging of edge cases - manual QC validation --- ### How `sdk.prs` fills this gap - **Standardizes variant matching** - matches SNPs by chromosome + position with fallbacks - filters valid genotypes and alleles - **Implements robust PRS computation** - calculates allele counts and weighted scores - aggregates scores consistently across variants - **Handles unmatched variants explicitly** - preserves all PGS rows for diagnostics - distinguishes matched vs non-matched SNPs - **Provides built-in QC checks** - detects missing matches - flags invalid weight distributions - identifies potential data issues (e.g., non-beta weights) - **Optimized for browser workflows** - avoids crashes from empty arrays or missing data - supports integration with cached PGS and 23andMe inputs --- #### Core PRS function `sdk.prs.Match3(mypgs, my23)` takes: - a PGS scoring file (`mypgs`) - a 23andMe genome file (`my23`) and computes a **Polygenic Risk Score (PRS)**. ### How it works: --- #### 1. Variant matching For each variant (SNP) in the PGS file: - match by **chromosome + position** (with rsID fallback when available) - filter genotypes containing the **effect allele** or **other allele** --- #### 2. Scoring matched variants For each matched SNP: - alleleCount ∈ {0, 1, 2} - effectWeight comes from the PGS file ![image@1.png](${await FileAttachment("image@1.png").url()}) --- #### 3. Tracking unmatched variants - **Match3 keeps all PGS rows** - matched → real scores - unmatched → `score = 0`, `status = "nomatch"` Outputs: - `data2.results` → all rows - `data2.pgsMatchMy23` → matched only --- #### 4. PRS calculation ![image.png](${await FileAttachment("image.png").url()}) --- #### 5. QC flags - No matches → `QC: false ("no matches")` - Any score > 100 → `QC: false ("large betas")` - All weights > −0.00002 → `QC: false ("not betas")` - Otherwise → `QC: true` --- ### `Match2` vs `MatchOptimized` | `Match2` | Matches every PGS variant by repeatedly scanning the entire 23andMe genome. It counts effect alleles, multiplies dosage by the variant weight, adds the contributions, exponentiates the sum, and performs basic QC. | | `MatchOptimized` | Performs the same matching, scoring, and QC, but first creates a chromosome-position hash index ([see speed gains](https://observablehq.com/@lorenasandoval88/prs-hash-index-optimization)). Each PGS variant can then be looked up directly instead of rescanning the genome. | **Main difference:** `Match2` has `O(n × m)` complexity, while `MatchOptimized` is approximately `O(n + m)`. For 313 PGS variants and 601,895 genome rows, this reduces the estimated work from **188,393,135** to **602,208 operations**. Both functions use only basic QC and do not fully check match coverage, duplicate conflicts, genome build, strand orientation, or exact allele compatibility. | Feature | `Match2` — old method | `MatchOptimized` — hash-index method | |---|---|---| | Purpose | Matches PGS and genome variants and calculates PRS. | Calculates the same PRS using faster matching. | | Matching | Scans the full genome for every PGS variant. | Indexes the genome once and directly looks up each variant. | | Scoring | Sums `effect-allele dosage × weight`, then exponentiates the sum. | Same calculation. | | QC | Checks invalid inputs, no matches, unusually large contributions, and weight type. | Same QC checks. | | Complexity | `O(n × m)` | Approximately `O(n + m)` | | Example work | `188,393,135` operations | `602,208` operations | | Main limitation | Very slow for large PGS models. | Uses more memory; allele, duplicate, build, and match-rate QC still need improvement. | | QC condition used by both functions | Result | What it actually checks | Important limitation | |---|---|---|---| | Invalid `mypgs.cols` | `PRS = "error"` and `QC = false` | Whether the PGS column definition exists and is an array. | Does not validate `mypgs.dt`, required columns, alleles, or weights. | | Invalid `my23.cols` | `PRS = "error"` and `QC = false` | Whether the genome column definition exists and is an array. | Does not validate `my23.dt` or required columns. | | `calcRiskScore.length === 0` | `QC = false` | Whether any variants were matched. | One match is enough to avoid this failure; no minimum coverage is required. | | `max(calcRiskScore) > 100` | `QC = false` | Whether an individual dosage-weight contribution exceeds `100`. | The message says “large betas,” but the code checks contributions, not the weights alone. Large negative contributions are not detected by this condition. | | `min(weights) > -0.00002` | `QC = false` | Whether all PGS weights are greater than approximately zero. | This is not a reliable way to determine whether weights are beta coefficients. A valid PGS containing only positive beta weights would fail. | | Otherwise | `QC = true` | Whether none of the preceding conditions were triggered. | Does not mean the PRS is fully validated or clinically interpretable. | | Missing recommended QC | Not implemented | Match coverage, genome build, exact allele compatibility, strand orientation, duplicate conflicts, missingness, and numeric-weight validation. | These checks should be added before treating the result as a reliable PRS. | --- ### Summary `MatchOptimized` provides a streamlined implementation for computing PRS from matched variants. 35:T42f,function toc(selector = "h2", heading = "Table of Contents") { return Generators.observe(notify => { let headings = []; function observed() { const h = Array.from(document.querySelectorAll(selector)); if (h.length !== headings.length || h.some((h, i) => headings[i] !== h)) { notify(html`${heading}

${DOM.text(h.textContent)}

`); } } const observer = new MutationObserver(observed); observer.observe(document.body, { childList: true, subtree: true }); observed(); return () => observer.disconnect(); }); }36:[{"id":"3bd2cf85bdbdd2466e5738e20a00ef6446891d228bed7ea45c138eac3706c1fbf3806b57a7108f9153682707acfb4d5434e790c8ea6108607162c9e4927eddc3","href":"https://static.observableusercontent.com/files/3bd2cf85bdbdd2466e5738e20a00ef6446891d228bed7ea45c138eac3706c1fbf3806b57a7108f9153682707acfb4d5434e790c8ea6108607162c9e4927eddc3","name":"image.png","type":"image/png","lastModified":1777404731026.741,"size":3468},{"id":"b5672f5031289beae63a669fdbd4dbeb724df89eb8e6100ae7823feb971c1ec36f232011fbc053d8270b142789e30595fcdac83ed42b9175b51c910fb492ea70","href":"https://static.observableusercontent.com/files/b5672f5031289beae63a669fdbd4dbeb724df89eb8e6100ae7823feb971c1ec36f232011fbc053d8270b142789e30595fcdac83ed42b9175b51c910fb492ea70","name":"image@1.png","type":"image/png","lastModified":1777404793602.076,"size":6086}] 2f:{"title":" Polygenic Risk Scores - SDK","cells":[{"id":432,"value":"# Polygenic Risk Scores - SDK\n\nBrowser-native JavaScript SDK for loading genomic [data](https://my.pgp-hms.org/public_genetic_data), polygenic score [files](https://www.pgscatalog.org/), PRS calculation and visualization.\nUI found [here](https://lorenasandoval88.github.io/polygenic_risk_scores/).\n","pinned":false,"hidden":false,"mode":"md"},{"id":30,"value":"sdk = await import (\"https://lorenasandoval88.github.io/diseasome/dist/sdk.mjs\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":688,"value":"toc()","pinned":false,"hidden":false,"mode":"ojs"},{"id":97,"value":"---","pinned":false,"hidden":false,"mode":"md"},{"id":449,"value":"## 1. `sdk.clustjs` — Clustering & Visualization Module","pinned":false,"hidden":false,"mode":"md"},{"id":452,"value":"$31","pinned":false,"hidden":false,"mode":"md"},{"id":483,"value":"### Example - Hclust svg","pinned":false,"hidden":false,"mode":"md"},{"id":612,"value":"example_plotting_data = sdk.clustjs.irisData","pinned":false,"hidden":false,"mode":"ojs"},{"id":459,"value":"//Example\nhclustSvg = await sdk.clustjs.hclust_plot({\n // data: matrix,\n // rowNames: users,\n // colNames: snps,\n width: 900,\n height: 400,\n clusterRows: clusterRows,\n clusterCols: clusterCols,\n clusteringDistanceRows: \"euclidean\",\n clusteringDistanceCols: \"euclidean\",\n clusteringMethodRows: \"complete\",\n clusteringMethodCols: \"complete\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":603,"value":"viewof clusterRows = Inputs.toggle({\n label: \"Cluster Rows\",\n value: true\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":606,"value":"viewof clusterCols = Inputs.toggle({\n label: \"Cluster Columns\",\n value: true\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":463,"value":"viewof plot1 = {\n const div = document.createElement(\"div\")\n const svg = hclustSvg\n div.appendChild(svg)\n return div\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":617,"value":"### Example - PCA svg","pinned":false,"hidden":false,"mode":"md"},{"id":621,"value":"//Example\npcaSvg = await sdk.clustjs.pca_plot({\n // data: matrix,\n // rowNames: users,\n // colNames: snps,\n width: 800,\n height: 200,\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":625,"value":"viewof plot2 = {\n const div = document.createElement(\"div\")\n const svg = pcaSvg\n div.appendChild(svg)\n return div\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":514,"value":"---","pinned":false,"hidden":false,"mode":"md"},{"id":55,"value":"## 2. `sdk.pgp` — Personal Genome Project Module","pinned":false,"hidden":false,"mode":"md"},{"id":477,"value":"$32","pinned":false,"hidden":false,"mode":"md"},{"id":480,"value":"Object.keys(sdk.pgp)","pinned":false,"hidden":false,"mode":"ojs"},{"id":505,"value":"### Examples","pinned":false,"hidden":false,"mode":"md"},{"id":428,"value":"all_data_types = sdk.pgp.fetchAvailableDataTypes()","pinned":false,"hidden":false,"mode":"ojs"},{"id":347,"value":"all_23andme_users = await sdk.pgp.allUsersMetaDataByType_fast(\"23andMe\") //23andme users","pinned":false,"hidden":false,"mode":"ojs"},{"id":904,"value":"all_health_rec_users = await sdk.pgp.allUsersMetaDataByType_fast(\"health records - CCR XML\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":907,"value":"healthIds = new Set(all_health_rec_users.map(d => d.id));","pinned":false,"hidden":false,"mode":"ojs"},{"id":910,"value":"overlap = all_23andme_users.filter(d => healthIds.has(d.id));","pinned":false,"hidden":false,"mode":"ojs"},{"id":524,"value":"cached in localforage under key, \"Genome:23andMe-allParticipants-fast\"","pinned":false,"hidden":false,"mode":"md"},{"id":841,"value":"The following require a user id, ie. \"hu50801B\" (Melinda) or \"huC8B936\" (James)","pinned":false,"hidden":false,"mode":"md"},{"id":62,"value":"user1_profile = await sdk.pgp.fetchProfile(\"hu50801B\")// Melinda","pinned":false,"hidden":false,"mode":"ojs"},{"id":942,"value":"user_huD96A97_profile = await sdk.pgp.fetchProfile(\"huD96A97\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":141,"value":"cached in localforage under key, \"Genome:23andMe-participant-hu416394\"","pinned":false,"hidden":false,"mode":"md"},{"id":936,"value":"filePath = all_23andme_users[0].downloadUrl","pinned":false,"hidden":false,"mode":"ojs"},{"id":124,"value":"user1_txt = sdk.pgp.get23Txt(filePath)","pinned":false,"hidden":false,"mode":"ojs"},{"id":67,"value":"cached in localforage under key, \"Genome:23andMe-txt-hu416394\"*","pinned":false,"hidden":false,"mode":"md"},{"id":883,"value":"### Use Cases","pinned":false,"hidden":false,"mode":"md"},{"id":833,"value":"\n Now, try filtering the 23andMe file for Y chromosome to check sex:\n \n \n 23andMe files may include Y chromosome positions even for XX (female) individuals; these typically appear as \"--\" (no call / missing data).\n \n \n","pinned":false,"hidden":false,"mode":"md"},{"id":854,"value":"inferSex(user1_txt)","pinned":false,"hidden":false,"mode":"ojs"},{"id":835,"value":"Y_rows = user1_txt.dt.filter(row => row[1] === \"Y\");","pinned":false,"hidden":false,"mode":"ojs"},{"id":851,"value":"function inferSex(parsed) {\n const chrIndex = parsed.cols.indexOf(\"chromosome\");\n const genoIndex = parsed.cols.indexOf(\"genotype\");\n\n const yRows = parsed.dt.filter(row => row[chrIndex] === \"Y\");\n const yCalled = yRows.filter(row => row[genoIndex] && row[genoIndex] !== \"--\");\n\n return {\n totalYRows: yRows.length,\n calledYRows: yCalled.length,\n sex: yCalled.length > 0 ? \"male (XY)\" : \"female (XX)\"\n };\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":860,"value":"\n Coarsely infer ancestry:\n \n Using data from the :contentReference[oaicite:0]{index=0}, ancestry can be approximated by comparing an individual’s genotypes at a small set of (~100) common SNPs to a global reference panel. Genotypes are encoded numerically (0, 1, 2), and similarity to reference populations is computed using distance-based methods or visualized via PCA, allowing the individual to be placed near major population clusters to estimate broad ancestry. *Does not capture fine-scale structure or precise admixture proportions.\n \n","pinned":false,"hidden":false,"mode":"md"},{"id":95,"value":"---","pinned":false,"hidden":false,"mode":"md"},{"id":41,"value":"## 3. `sdk.pgs` — PGS Catalog Module","pinned":false,"hidden":false,"mode":"md"},{"id":530,"value":"$33","pinned":false,"hidden":false,"mode":"md"},{"id":536,"value":"### Examples","pinned":false,"hidden":false,"mode":"md"},{"id":162,"value":"pgs1_txt = sdk.pgs.getPgsTxt([\"PGS000001\"])","pinned":false,"hidden":false,"mode":"ojs"},{"id":561,"value":"*cached in localforage under key, \"PGS_Catalog:id-PGS000001\"*","pinned":false,"hidden":false,"mode":"md"},{"id":23,"value":"all_pgs_scores = await sdk.pgs.fetchAllScores()","pinned":false,"hidden":false,"mode":"ojs"},{"id":75,"value":"*cached in localforage under key, \"PGS_Catalog:all-score-summary\"*","pinned":false,"hidden":false,"mode":"md"},{"id":580,"value":"---","pinned":false,"hidden":false,"mode":"md"},{"id":153,"value":"## 4. `sdk.prs` — PRS Matching & Calculation\n","pinned":false,"hidden":false,"mode":"md"},{"id":567,"value":"$34","pinned":false,"hidden":false,"mode":"md"},{"id":770,"value":"### Example","pinned":false,"hidden":false,"mode":"md"},{"id":953,"value":"polygenic_risk_score_results_FAST = sdk.prs.MatchOptimized(pgs1_txt[0],user1_txt)","pinned":true,"hidden":false,"mode":"ojs"},{"id":582,"value":"*cached in localforage under key, \"PRS: hu09B28E_PGS000001\"*","pinned":false,"hidden":false,"mode":"md"},{"id":722,"value":"---","pinned":false,"hidden":false,"mode":"md"},{"id":725,"value":"# Helper functions\n","pinned":false,"hidden":false,"mode":"md"},{"id":714,"value":"$35","pinned":false,"hidden":false,"mode":"ojs"},{"id":824,"value":" function oldToc(selector = \"h2,h3,h4\", heading = \"Table of Contents\") {\n return Generators.observe(notify => {\n let headings = [];\n\n function observed() {\n const h = Array.from(document.querySelectorAll(selector));\n\n if (h.length !== headings.length || h.some((h, i) => headings[i] !== h)) {\n notify(html`${heading}
${Array.from(headings = h, h => {\n const indent =\n h.tagName === \"H3\" ? \"24px\" :\n h.tagName === \"H4\" ? \"48px\" :\n \"0px\";\n\n return Object.assign(\n html`
\n ${DOM.text(h.textContent)}\n
`,\n { onclick: e => (e.preventDefault(), h.scrollIntoView()) }\n );\n })}
`);\n }\n }\n\n const observer = new MutationObserver(observed);\n observer.observe(document.body, { childList: true, subtree: true });\n observed();\n return () => observer.disconnect();\n });\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":919,"value":"vcfUrl = \"https://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20130502/ALL.chr22.phase3_shapeit2_mvncall_integrated_v5b.20130502.genotypes.vcf.gz\"\n","pinned":true,"hidden":false,"mode":"ojs"}],"files":"$W36","stdlib":"1","titling":"auto"}