32:T977,vtt_string = `WEBVTT Kind: captions Language: en 00:00:00.176 --> 00:00:01.716 >> We have recently identified 00:00:01.716 --> 00:00:04.816 that some astronauts experience changes in their vision, 00:00:04.816 --> 00:00:07.756 which might be related to the cardiovascular system. 00:00:07.756 --> 00:00:10.986 Our hearts pump blood around our body through blood vessels. 00:00:10.986 --> 00:00:13.016 Special adaptations in our bodies ensure 00:00:13.016 --> 00:00:18.246 that fluid is evenly distributed despite the pull of gravity. 00:00:18.246 --> 00:00:21.206 In space, astronauts no longer experience the downward pull 00:00:21.206 --> 00:00:23.446 of gravity, and the fluid in their bodies tends 00:00:23.446 --> 00:00:26.426 to move towards the upper body and the head. 00:00:26.426 --> 00:00:28.376 Our cranium is a rigid container. 00:00:28.376 --> 00:00:30.126 As the fluid moves towards the head, 00:00:30.126 --> 00:00:32.546 it causes the pressure inside the skull to rise. 00:00:32.546 --> 00:00:35.146 This is known as increased intracranial pressure. 00:00:35.146 --> 00:00:38.376 The optic nerve travels from the brain to the eye. 00:00:38.466 --> 00:00:41.006 The increased pressure from the cranium travels 00:00:41.006 --> 00:00:42.486 down the nerve and affects the eye. 00:00:42.486 --> 00:00:46.196 It causes the optic nerve to be squeezed and the optic disc 00:00:46.246 --> 00:00:48.366 where the optic nerve meets the eye to swell. 00:00:48.366 --> 00:00:51.556 The back of the eyeball flattens as pressure builds behind it, 00:00:51.556 --> 00:00:54.446 and the blood vessels in the back of the eyeball also swell. 00:00:54.446 --> 00:00:58.086 These changes can affect the astronaut's vision. 00:00:58.086 --> 00:01:00.956 One change is that the astronauts become far sighted. 00:01:00.956 --> 00:01:03.646 A second change is that they can develop cotton wool spots, 00:01:03.646 --> 00:01:06.126 which represent damage of the nerve fibers 00:01:06.126 --> 00:01:07.046 in the back of the eye. 00:01:07.046 --> 00:01:10.196 These can cause discrete areas of vision loss. 00:01:10.196 --> 00:01:14.286 NASA is working to understand to prevent these changes 00:01:14.286 --> 00:01:16.576 in astronauts which may also help us understand 00:01:16.576 --> 00:01:22.096 to prevent related vision changes here on Earth. ------------------------------688ed48833b8-- `33:[] 30:{"title":"Working with data from transcripts II - now includes visualization example","cells":[{"id":0,"value":"md`# Working with data from transcripts II - now includes visualization example`","pinned":false,"hidden":false,"mode":"ojs"},{"id":182,"value":"md `## \"vtt-to-json\" for vtt file format to json conversion & \"compromise\" for basic NLP`","pinned":false,"hidden":false,"mode":"ojs"},{"id":166,"value":"md `### Why?`","pinned":false,"hidden":false,"mode":"ojs"},{"id":168,"value":"md `Watching videos or listening to audio is slow and requires a human, a costly human. As speech-to-text gets better transcription is less costly and more common. Data visualizations of transcripts, or keywords, or attributes of groups of words should be faster for many search/compare/understand tasks than watching or listening at real-time speeds.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":173,"value":"md `### What is in this notebook?`","pinned":false,"hidden":false,"mode":"ojs"},{"id":159,"value":"md `This is a quick demo of data conversion before you would do data visuailzation. It show how you might use vtt-to-json for converting vtt files into easier to work with json and compromise for basic natural language processing as early steps before doing data visuaization.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":163,"value":"md `VTT is a web format standard for video transcripts.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":72,"value":"md `### Import the vtt-to-json function`","pinned":false,"hidden":false,"mode":"ojs"},{"id":2,"value":"vttToJson = require('vtt-to-json@0.1.1/index.js').catch(() => window.convertVttToJson)","pinned":true,"hidden":false,"mode":"ojs"},{"id":89,"value":"md `We'll use the transcript file from this youtube video on \"Astronaut Vision Changes in Microgravity\" for demonstration purposes. Scroll down to the bottom of the transcript for the next part.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":189,"value":"//// Uncomment the lines below to see video!\nhtml``","pinned":false,"hidden":false,"mode":"ojs"},{"id":198,"value":"md `#### Paste in the vtt file as a string`","pinned":false,"hidden":false,"mode":"ojs"},{"id":291,"value":"ytdlcore = require('https://bundle.run/ytdl-core@0.20.2')","pinned":false,"hidden":false,"mode":"ojs"},{"id":235,"value":"videoID = \"q_q61B-DyPk\"","pinned":true,"hidden":false,"mode":"ojs"},{"id":241,"value":"getYoutubeSubtitles = require('@joegesualdo/get-youtube-subtitles-node@0.1.0/index.js').catch(() => window.getYoutubeSubtitles) ","pinned":false,"hidden":false,"mode":"ojs"},{"id":266,"value":"function getSubtitles(videoID){\n getYoutubeSubtitles(videoID).then(subtitles => {\n console.log(subtitles)\n subtitles_holder.push(subtitles)\n return subtitles\n})\n.catch(err => {\n console.log(err)\n subtitles_holder.push(err)\n})\n}\n","pinned":true,"hidden":false,"mode":"ojs"},{"id":277,"value":"// subtitles = getSubtitles(videoID)","pinned":true,"hidden":false,"mode":"ojs"},{"id":269,"value":"subtitles_holder = [\"\"]","pinned":false,"hidden":false,"mode":"ojs"},{"id":43,"value":"$32","pinned":false,"hidden":false,"mode":"ojs"},{"id":93,"value":"md `### Use the vttToJson function we established above to return a JSON of the VTT file to make it easier to work with`","pinned":false,"hidden":false,"mode":"ojs"},{"id":66,"value":"json_transcript_output = vttToJson(vtt_string)\n.then((result) => {\n return(result)\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":95,"value":"md `Now that the data is in json format, it is a little easier to work with. Click on the arrow next to \"Array\" to expand the json and see the key/value pairs.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":98,"value":"md `## We can bring in the compromise JavaScript NLP library to do some quick analysis on the words in the transcript`","pinned":false,"hidden":false,"mode":"ojs"},{"id":139,"value":"md `This is a good demo of the compromise library in Observable.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":142,"value":"md `### Importing the compromise library with require`","pinned":false,"hidden":false,"mode":"ojs"},{"id":103,"value":"nlp =require('compromise')","pinned":false,"hidden":false,"mode":"ojs"},{"id":145,"value":"md `\"getAllWords\" is a function for getting all the words from the JSON created from the vtt transcript with vtt-to-json.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":105,"value":"function getAllWords(json_){\n let all_words = [];\n for (var i = 0; i < json_.length; i++) { \n \tlet words = json_[i][\"words\"]\n \tfor (var j = 0; j < words.length; j++){\n all_words.push(words[j][\"word\"])\n }\t\n\t}\n return all_words\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":108,"value":"all_words_array = getAllWords(json_transcript_output)","pinned":false,"hidden":false,"mode":"ojs"},{"id":149,"value":"md `\"turnArrayofWordsIntoString\" is a function that takes the array of words found above and turns it into a string with no commas, periods, etc.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":115,"value":"function turnArrayofWordsIntoString(arrayOfWords){\n\tlet wordString = \"\";\n \tfor (var i = 0; i < arrayOfWords.length; i++) {\n \tconsole.log(arrayOfWords[i])\n \twordString = wordString+\" \"+arrayOfWords[i]\n }\n return wordString\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":201,"value":"md `Get all the words`","pinned":false,"hidden":false,"mode":"ojs"},{"id":118,"value":"words = turnArrayofWordsIntoString(all_words_array)","pinned":true,"hidden":false,"mode":"ojs"},{"id":343,"value":"words","pinned":false,"hidden":false,"mode":"ojs"},{"id":151,"value":"md `We can then give that string of words to compromise-nlp and ask it to give us arrays of words back according to parts of speech, like nouns or verbs. `","pinned":false,"hidden":false,"mode":"ojs"},{"id":203,"value":"md `Get all the verbs.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":128,"value":"VERBS = nlp(words).verbs().out('array')","pinned":true,"hidden":false,"mode":"ojs"},{"id":205,"value":"md `Get all the nouns.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":134,"value":"NOUNS = nlp(words).nouns().out('array')","pinned":true,"hidden":false,"mode":"ojs"},{"id":216,"value":"viewof min_length_of_word = html``","pinned":false,"hidden":false,"mode":"ojs"},{"id":222,"value":"min_length_of_word","pinned":false,"hidden":false,"mode":"ojs"},{"id":207,"value":"function longerWords(words){\n\tlet moreImpWords = []\n \tfor (let eachW in words){\n \tif (words[eachW].length > min_length_of_word){\n moreImpWords.push(words[eachW])\n }\n \telse if (words[eachW].includes(\" \")){\n moreImpWords.push(words[eachW])\n }\n }\n return moreImpWords\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":230,"value":"md `Subset of NOUNS: includes words with spaces, so multiple words, or words with characters > than min_length_of_word variable`","pinned":false,"hidden":false,"mode":"ojs"},{"id":211,"value":"longer_nouns = longerWords(NOUNS)","pinned":false,"hidden":false,"mode":"ojs"},{"id":306,"value":"md `# We can also look at the words over time`","pinned":false,"hidden":false,"mode":"ojs"},{"id":313,"value":"md `First, lets go back to the json object we got that had both the times and words`","pinned":false,"hidden":false,"mode":"ojs"},{"id":315,"value":"json_transcript_output","pinned":true,"hidden":false,"mode":"ojs"},{"id":317,"value":"function getNounsByTime(words_andtime){\n var newArrayForObj = []\n for(var i = 0; i < words_andtime.length; i++){\n var newObj = {}\n newObj['start'] = json_transcript_output[i]['start']\n newObj['end'] = json_transcript_output[i]['end']\n var nouns_obj = json_transcript_output[i]['words']\n var all_nouns = []\n for (var j = 0; j < nouns_obj.length; j++){\n all_nouns.push(nouns_obj[j]['word'])\n }\n newObj['nouns'] = longerWords2(all_nouns)\n //newObj['nouns'] = nlp(turnArrayofWordsIntoString(json_transcript_output[i]['words'])).nouns().out('array')\n newArrayForObj.push(newObj)\n }\n return newArrayForObj\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":324,"value":"nouns_and_timeGroup = getNounsByTime(json_transcript_output)","pinned":false,"hidden":false,"mode":"ojs"},{"id":352,"value":"function reStructureWordsForTime(nouns_and_timeGroup){\n var newArrayofObj = []\n for(var i = 0; i < nouns_and_timeGroup.length; i++){\n var start = nouns_and_timeGroup[i]['start']\n for (var j = 0; j < nouns_and_timeGroup[i]['nouns'].length; j++){\n var wordObj = {}\n wordObj['time'] = parseInt(start)+(j*200)\n wordObj['word'] = nouns_and_timeGroup[i]['nouns'][j]\n newArrayofObj.push(wordObj)\n }\n }\n return newArrayofObj\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":363,"value":"flatWordArray = reStructureWordsForTime(nouns_and_timeGroup)","pinned":false,"hidden":false,"mode":"ojs"},{"id":370,"value":"function changeDataStructureToMatch(data){\n var new_arrayObj = []\n for(var i = 0; i < data.length; i++){\n var newObj = {}\n newObj['y'] = i\n newObj['x'] = data[i]['time']\n newObj['name'] = data[i]['word']\n new_arrayObj.push(newObj)\n }\n return new_arrayObj\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":372,"value":"data = changeDataStructureToMatch(flatWordArray)","pinned":false,"hidden":false,"mode":"ojs"},{"id":402,"value":"function longerWords2(words){\n\tlet moreImpWords = []\n \tfor (let eachW in words){\n \tif (words[eachW].length > min_length_of_word2){\n moreImpWords.push(words[eachW])\n }\n \telse if (words[eachW].includes(\" \")){\n moreImpWords.push(words[eachW])\n }\n }\n return moreImpWords\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":399,"value":"viewof min_length_of_word2 = html``","pinned":false,"hidden":false,"mode":"ojs"},{"id":408,"value":"min_length_of_word2","pinned":false,"hidden":false,"mode":"ojs"},{"id":378,"value":"md `## Now lets visualize all the nouns in a short video but change the length of the included words`","pinned":false,"hidden":false,"mode":"ojs"},{"id":380,"value":"chart = {\n const svg = d3.select(DOM.svg(width, height));\n \n svg.append(\"g\")\n .call(xAxis);\n \n svg.append(\"g\")\n .call(yAxis);\n \n svg.append(\"g\")\n .attr(\"stroke\", \"steelblue\")\n .attr(\"stroke-width\", 1.5)\n .attr(\"fill\", \"none\")\n .selectAll(\"circle\")\n .data(data)\n .enter().append(\"circle\")\n .attr(\"cx\", d => x(d.x))\n .attr(\"cy\", d => y(d.y))\n .attr(\"r\", 2);\n \n svg.append(\"g\")\n .attr(\"font-family\", \"sans-serif\")\n .attr(\"font-size\", 12)\n .selectAll(\"text\")\n .data(data)\n .enter().append(\"text\")\n .attr(\"x\", d => x(d.x) + 7)\n .attr(\"y\", d => y(d.y))\n .attr(\"dy\", \"0.35em\")\n .text(d => d.name);\n \n return svg.node();\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":381,"value":"height = 600","pinned":false,"hidden":false,"mode":"ojs"},{"id":383,"value":"margin = ({top: 20, right: 30, bottom: 30, left: 40})","pinned":false,"hidden":false,"mode":"ojs"},{"id":386,"value":"x = d3.scaleLinear()\n .domain(d3.extent(data, d => d.x)).nice()\n .range([margin.left, width - margin.right])","pinned":false,"hidden":false,"mode":"ojs"},{"id":387,"value":"y = d3.scaleLinear()\n .domain(d3.extent(data, d => d.y)).nice()\n .range([height - margin.bottom, margin.top])","pinned":false,"hidden":false,"mode":"ojs"},{"id":389,"value":"xAxis = g => g\n .attr(\"transform\", `translate(0,${height - margin.bottom})`)\n .call(d3.axisBottom(x))\n .call(g => g.select(\".domain\").remove())\n .call(g => g.append(\"text\")\n .attr(\"x\", width - margin.right)\n .attr(\"y\", -4)\n .attr(\"fill\", \"#000\")\n .attr(\"font-weight\", \"bold\")\n .attr(\"text-anchor\", \"end\")\n .text(data.x))","pinned":false,"hidden":false,"mode":"ojs"},{"id":391,"value":"yAxis = g => g\n .attr(\"transform\", `translate(${margin.left},0)`)\n .call(d3.axisLeft(y))\n .call(g => g.select(\".domain\").remove())\n .call(g => g.select(\".tick:last-of-type text\").clone()\n .attr(\"x\", 4)\n .attr(\"text-anchor\", \"start\")\n .attr(\"font-weight\", \"bold\")\n .text(data.y))","pinned":false,"hidden":false,"mode":"ojs"},{"id":393,"value":"d3 = require(\"d3@5\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":416,"value":"md `## Listening is slow. Reading full transcripts is slow. How can we pull out the more powerful representative bits and arrange them visually to speed insight!`","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W33","stdlib":"1","titling":"auto"}