30:T644,We'll use several different sources for data and ontology throughout the **kglab** tutorial, although most of it focuses on progressive examples that use [*FoodOn*](https://www.nature.com/articles/s41538-018-0032-6). *FoodOn* – subtitled "a farm to fork ontology" – takes a comprehensive view of the data and metadata involved in our food supply, beginning with seed genomics, micronutrients, the biology of food alergies, etc. This work is predicated on leveraging large knowledge graphs to represent the different areas of science, technology, business, public policy, etc.: > The need to represent knowledge about food is central to many human activities including agriculture, medicine, food safety inspection, shopping patterns, and sustainable development. FoodOn is an ontology – a controlled vocabulary which can be used by both people and computers – to name all parts of animals, plants, and fungai which can bear a food role for humans and domesticated animals, as well as derived food products and the processes used to make them. For more details, see: * * * * For primary sources, see: [[vardeman2014ceur]](https://derwen.ai/docs/kgl/biblio/#vardeman2014ceur), [[sam2014odp]](https://derwen.ai/docs/kgl/biblio/#sam2014odp), [[dooley2018npj]](https://derwen.ai/docs/kgl/biblio/#dooley2018npj), [[hitzler2018]](https://derwen.ai/docs/kgl/biblio/#hitzler2018)31:T496,One frequently asked question is about the size of the graphs that we're using in the **kglab** tutorial. The short answer: "No, these aren't trivial graphs." We'll start out with small examples, to show the basics for how to construct an RDF graph. Most of the examples here will use a knowledge graph with ~300 nodes and ~2000 edges. This is a *non-trivial* size, especially when you start working with some graph algorithms. Again, this tutorial has learning as its main intent, and this size of graph is ideal for running queries, validation, graph algorithms, visualization, etc., with the kinds of compute and memory resources available on contemporary laptops. In other words, we prioritize datasets that are large enough for examples to illustrate common use cases, though small enough for learners to understand. * 10^6 or more nodes are needed for deep learning * 10^8 can run on contemporary laptops * larger graphs require hardware accelerators (e.g., GPUs) or cloud-based clusters The full `recipes.tsv` dataset includes nearly 250,000 recipes. In some of the later examples, we'll work with that entire dataset – which is definitely non-trivial.32:T135c,/* these are dependencies that were built in pyodide docker for kglab in case they could not be installed as wheel packages */ otherKglabDependencies = Promise.all([ await pyodide.loadPackage( "https://thadk.net/pyodide_packages/pyodide_18_1/CLAPACK.js" ), await pyodide.loadPackage( "https://thadk.net/pyodide_packages/pyodide_18_1/asttokens.js" ), await pyodide.loadPackage( "https://thadk.net/pyodide_packages/pyodide_18_1/chocolate.js" ), await pyodide.loadPackage( "https://thadk.net/pyodide_packages/pyodide_18_1/colorama.js" ), await pyodide.loadPackage( "https://thadk.net/pyodide_packages/pyodide_18_1/csvwlib.js" ), // await pyodide.loadPackage( // "https://thadk.net/pyodide_packages/pyodide_18_1/cycler.js" // ), await pyodide.loadPackage( "https://thadk.net/pyodide_packages/pyodide_18_1/dataclasses.js" ), await pyodide.loadPackage( "https://thadk.net/pyodide_packages/pyodide_18_1/decorator.js" ), await pyodide.loadPackage( "https://thadk.net/pyodide_packages/pyodide_18_1/executing.js" ), await pyodide.loadPackage( "https://thadk.net/pyodide_packages/pyodide_18_1/fsspec.js" ), await pyodide.loadPackage( "https://thadk.net/pyodide_packages/pyodide_18_1/gcsfs.js" ), // await pyodide.loadPackage( // "https://thadk.net/pyodide_packages/pyodide_18_1/gensim.js" // ), await pyodide.loadPackage( "https://thadk.net/pyodide_packages/pyodide_18_1/html5lib.js" ), await pyodide.loadPackage( "https://thadk.net/pyodide_packages/pyodide_18_1/icecream.js" ), await pyodide.loadPackage( "https://thadk.net/pyodide_packages/pyodide_18_1/isodate.js" ), // await pyodide.loadPackage( // "https://thadk.net/pyodide_packages/pyodide_18_1/joblib.js" // ), await pyodide.loadPackage( "https://thadk.net/pyodide_packages/pyodide_18_1/keepalive.js" ), // await pyodide.loadPackage("https://thadk.net/pyodide_packages/pyodide_18_1/kglab.js"), // await pyodide.loadPackage( // "https://thadk.net/pyodide_packages/pyodide_18_1/kiwisolver.js" // ), // await pyodide.loadPackage("https://thadk.net/pyodide_packages/pyodide_18_1/matplotlib.js"), // await pyodide.loadPackage( // "https://thadk.net/pyodide_packages/pyodide_18_1/micropip.js" // ), // await pyodide.loadPackage("https://thadk.net/pyodide_packages/pyodide_18_1/networkx.js"), // await pyodide.loadPackage("https://thadk.net/pyodide_packages/pyodide_18_1/numpy.js"), await pyodide.loadPackage( "https://thadk.net/pyodide_packages/pyodide_18_1/owlrl.js" ), // await pyodide.loadPackage( // "https://thadk.net/pyodide_packages/pyodide_18_1/packaging.js" // ), // await pyodide.loadPackage("https://thadk.net/pyodide_packages/pyodide_18_1/pandas.js"), // await pyodide.loadPackage( // "https://thadk.net/pyodide_packages/pyodide_18_1/patsy.js" // ), // await pyodide.loadPackage("https://thadk.net/pyodide_packages/pyodide_18_1/pillow.js"), await pyodide.loadPackage( "https://thadk.net/pyodide_packages/pyodide_18_1/pygments.js" ), await pyodide.loadPackage( "https://thadk.net/pyodide_packages/pyodide_18_1/pylev.js" ), await pyodide.loadPackage( "https://thadk.net/pyodide_packages/pyodide_18_1/pynvml.js" ), // await pyodide.loadPackage( // "https://thadk.net/pyodide_packages/pyodide_18_1/pyparsing.js" // ), await pyodide.loadPackage( "https://thadk.net/pyodide_packages/pyodide_18_1/pyshacl.js" ), // await pyodide.loadPackage( // "https://thadk.net/pyodide_packages/pyodide_18_1/python-dateutil.js" // ), // await pyodide.loadPackage( // "https://thadk.net/pyodide_packages/pyodide_18_1/pytz.js" // ), await pyodide.loadPackage( "https://thadk.net/pyodide_packages/pyodide_18_1/rdflib.js" ), await pyodide.loadPackage( "https://thadk.net/pyodide_packages/pyodide_18_1/requests.js" ), // await pyodide.loadPackage( // "https://thadk.net/pyodide_packages/pyodide_18_1/scikit-learn.js" // ), // await pyodide.loadPackage( // "https://thadk.net/pyodide_packages/pyodide_18_1/scipy.js" // ), // await pyodide.loadPackage( // "https://thadk.net/pyodide_packages/pyodide_18_1/setuptools.js" // ), // await pyodide.loadPackage( // "https://thadk.net/pyodide_packages/pyodide_18_1/six.js" // ), await pyodide.loadPackage( "https://thadk.net/pyodide_packages/pyodide_18_1/smart_open.js" ), await pyodide.loadPackage( "https://thadk.net/pyodide_packages/pyodide_18_1/sparqlwrapper.js" ), // await pyodide.loadPackage( // "https://thadk.net/pyodide_packages/pyodide_18_1/statsmodels.js" // ), await pyodide.loadPackage( "https://thadk.net/pyodide_packages/pyodide_18_1/tqdm.js" ), await pyodide.loadPackage( "https://thadk.net/pyodide_packages/pyodide_18_1/urlpath.js" ), await pyodide.loadPackage( "https://thadk.net/pyodide_packages/pyodide_18_1/webencodings.js" ) ])33:T5c6,// Micropip dependencies are simple "wheel" python packages that do not need to be built in any special way. You just have to use micropip with a PyPi wheel URL or package name. micropipDependenciesPy = py`import micropip, os # Github Pages does not succeed, perhaps mime types, only raw github. micropip.install("https://files.pythonhosted.org/packages/7c/f7/280bf4a3830cf6723535096342785aa3a9a15251b188b6bc592da2dda53f/pslpython-2.2.2-py3-none-any.whl") micropip.install("https://files.pythonhosted.org/packages/af/f4/524415c0744552cce7d8bf3669af78e8a069514405ea4fcbd0cc44733744/urllib3-1.26.7-py2.py3-none-any.whl") micropip.install("https://files.pythonhosted.org/packages/47/84/b06f6729fac8108c5fa3e13cde19b0b3de66ba5538c325496dbe39f5ff8e/charset_normalizer-2.0.9-py3-none-any.whl") micropip.install("https://files.pythonhosted.org/packages/37/45/946c02767aabb873146011e665728b680884cd8fe70dde973c640e45b775/certifi-2021.10.8-py2.py3-none-any.whl") micropip.install("https://files.pythonhosted.org/packages/04/a2/d918dcd22354d8958fe113e1a3630137e0fc8b44859ade3063982eacd2a4/idna-3.3-py3-none-any.whl") micropip.install("language-tags") micropip.install("uritemplate") #micropip.install("https://files.pythonhosted.org/packages/07/d1/e87844ec86e96df7364f21af2263ad6030c0d727660ae89935c7af56a540/pyvis-0.1.9-py3-none-any.whl") micropip.install("https://thadk.net/pyodide_packages/pyodide_18_1/pyvis-0.1.9-py2.py3-none-any.whl") os.listdir("/lib/python3.9/site-packages/") `34:[{"id":"ece1832016566b42c8d77e58bf6acd80bbfd1635e2fb7c60c98e4d16e8b26c6569820229fec0c87e1505451c02262727c7f20032c3bdd12cc610aa3a5a93e0f3","href":"https://static.observableusercontent.com/files/ece1832016566b42c8d77e58bf6acd80bbfd1635e2fb7c60c98e4d16e8b26c6569820229fec0c87e1505451c02262727c7f20032c3bdd12cc610aa3a5a93e0f3","name":"kglab-main.zip","type":"application/zip","lastModified":1640308241435.016,"size":17569732}] 2e:{"title":"ex0_0 Data Sources","cells":[{"id":0,"value":"# ex0_0 Data Sources\n\nThroughout this tutorial we'll work with data in the `dat` subdirectory:","pinned":false,"hidden":false,"mode":"md"},{"id":287,"value":"kgzip.filenames.filter((a) => a.indexOf(\"kglab-main/dat\") !== -1)","pinned":true,"hidden":false,"mode":"ojs"},{"id":281,"value":"In particular, we'll work with a series of *progressive examples* based on the \n`dat/recipes.csv` CSV file.\nThis data comes from a \n[Kaggle dataset](https://www.kaggle.com/shuyangli94/food-com-recipes-and-user-interactions/metadata)\nthat describes metadata about [Food.com](https://food.com/):\n\n> \"Food.com Recipes and Interactions\" \nShuyang Li \nKaggle (2019) \n","pinned":false,"hidden":false,"mode":"md"},{"id":221,"value":"Hover your mouse pointer over the nodes to see the popup titles.\nAlso, try dragging a node to a different location to see how the graph reshapes dynamically.","pinned":false,"hidden":false,"mode":"md"},{"id":291,"value":"One of the simpler recipes in that dataset is `\"anytime crepes\"` at \n\n * id: 327593\n * minutes: 8\n * ingredients: `\"['egg', 'milk', 'whole wheat flour']\"`\n \nThe tutorial begins by showing how to represent the metadata for this recipe in a knowledge graph, then gradually builds up more and more information about this collection of recipes.\n\nTo start, let's load and examine the CSV data:","pinned":false,"hidden":false,"mode":"md"},{"id":295,"value":"df = py`\nfrom os.path import dirname\nimport os\nimport io\n\nbuffer = io.StringIO(${await kgzip.file(\"kglab-main/dat/recipes.csv\").text()})\ndf = ${pd}.read_csv(buffer)\ndf\n`","pinned":true,"hidden":false,"mode":"ojs"},{"id":319,"value":"table(df.to_dict(\"records\").toJs().map(Object.fromEntries))","pinned":true,"hidden":false,"mode":"ojs"},{"id":297,"value":"Now let's drill down to the metadata for the `\"anytime crepes\"` recipe","pinned":false,"hidden":false,"mode":"md"},{"id":300,"value":"recipe_row = py`\ndf = ${df}\nrecipe_row = df[df[\"name\"] == \"anytime crepes\"].iloc[0]\nrecipe_row`","pinned":true,"hidden":false,"mode":"ojs"},{"id":321,"value":"recipe_row.to_string()","pinned":false,"hidden":false,"mode":"ojs"},{"id":330,"value":"Given that we have a rich source of *linked data* to use, next we need to focus on *knowledge representation*. \nWe'll use the [FoodOn](https://foodon.org/design/foodon-relations/) ontology (see below) to represent recipes, making use of two of its *controlled vocabularies*:\n\n * \n * \n \nThe first one defines an entity called `Recipe` which has the full URL of and we'll use that to represent our recipe data from the *Food.com* dataset.\n \nIt's a common practice to abbreviate the first part of the URL for a controlled vocabulary with a *prefix*.\nIn this case we'll use the prefix conventions used in previous publications related to this ontology:\n\n| URL | prefix | \n| --- | --- | \n| | `wtm:` |\n| | `ind:` |","pinned":false,"hidden":false,"mode":"md"},{"id":334,"value":"Now let's represent the data using this ontology, starting with the three ingredients for the **anytime crepes** recipe:","pinned":false,"hidden":false,"mode":"md"},{"id":338,"value":"py`ingredients = eval(${recipe_row}[\"ingredients\"])\ningredients`","pinned":true,"hidden":false,"mode":"ojs"},{"id":341,"value":"These ingredients become represented, respectively, as:\n\n * `ind:ChickenEgg`\n * `ind:CowMilk`\n * `ind:WholeWheatFlour`","pinned":false,"hidden":false,"mode":"md"},{"id":345,"value":"## Ontology Sources","pinned":false,"hidden":false,"mode":"md"},{"id":348,"value":"$30","pinned":false,"hidden":false,"mode":"md"},{"id":355,"value":"We'll work through several examples of representation, although here's an example of what a full recipe in *FoodOn* would look like:","pinned":false,"hidden":false,"mode":"md"},{"id":361,"value":"`owl:NamedIndividual a wtm:Recipe ;\n rdf:about ind:BananaBlueberryAlmondFlourMuffin ;\n wtm:hasIngredient ind:AlmondMeal ;\n wtm:hasIngredient ind:AppleCiderVinegar ;\n wtm:hasIngredient ind:BakingSoda ;\n wtm:hasIngredient ind:Banana ;\n wtm:hasIngredient ind:Blueberry ;\n wtm:hasIngredient ind:ChickenEgg ;\n wtm:hasIngredient ind:Honey ;\n wtm:isRecommendedForCourse wtm:Dessert ;\n wtm:isRecommendedForMeal wtm:Breakfast ;\n wtm:isRecommendedForMeal wtm:Snack ;\n wtm:hasCookTime \"PT60M\"^^xsd:duration ;\n wtm:hasCookingTemperature \"350\"^^xsd:integer ;\n wtm:serves \"4\"^^xsd:integer ;\n rdfs:label \"banana blueberry almond flour muffin\" ;\n skos:definition \"a banana blueberry muffin made with almond flour\" ;\n skos:scopeNote \"recipe\" ;\n prov:wasDerivedFrom \n.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":364,"value":"## Graph Size Comparisons","pinned":false,"hidden":false,"mode":"md"},{"id":367,"value":"$31","pinned":false,"hidden":false,"mode":"md"},{"id":225,"value":"---","pinned":false,"hidden":false,"mode":"md"},{"id":215,"value":"## Dependencies","pinned":false,"hidden":false,"mode":"md"},{"id":315,"value":"import { table } from \"@gnestor/table\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":5,"value":"/* these are dependencies of kglab that are built into the standard pyodide scientific stack */\npackagesPy = await pyodide.loadPackage([\n \"numpy\",\n \"micropip\",\n \"six\",\n \"scipy\",\n \"pyparsing\",\n \"scikit-learn\",\n \"pandas\",\n \"networkx\",\n \"pillow\",\n \"matplotlib\",\n \"statsmodels\",\n \"urllib3\"\n])","pinned":false,"hidden":false,"mode":"ojs"},{"id":7,"value":"kglabDependency = {\n console.log(packagesPy);\n // return await pyodide.loadPackage(\"http://localhost:8000/ete3.js\");\n return await pyodide.loadPackage(\n \"https://thadk.net/pyodide_packages/pyodide_18_1/kglab.js\"\n );\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":17,"value":"$32","pinned":false,"hidden":false,"mode":"ojs"},{"id":12,"value":"$33","pinned":false,"hidden":false,"mode":"ojs"},{"id":305,"value":"pd = py`\nimport pandas as pd#${packagesPy}\npd`","pinned":true,"hidden":false,"mode":"ojs"},{"id":42,"value":"kglab = py`\n\n# make completely sure pyvis got included\nimport micropip\nmicropip.install(\"https://thadk.net/pyodide_packages/pyodide_18_1/pyvis-0.1.9-py2.py3-none-any.whl\")\n\nimport kglab\n# ${otherKglabDependencies} ${kglabDependency} ${micropipDependenciesPy}\nkglab\n`","pinned":true,"hidden":false,"mode":"ojs"},{"id":182,"value":"py`dir(${kglab})`","pinned":true,"hidden":false,"mode":"ojs"},{"id":185,"value":"py`repr(${kglab})`","pinned":true,"hidden":false,"mode":"ojs"},{"id":82,"value":"kgzip = FileAttachment(\"kglab-main.zip\").zip()","pinned":false,"hidden":false,"mode":"ojs"},{"id":154,"value":"import { py, pyodide, downloadPyoFile } from \"@thadk/pyodide-18\"","pinned":true,"hidden":false,"mode":"ojs"}],"files":"$W34","stdlib":"1","titling":"auto"}