31:[] 2f:{"title":"Wikification Part II: How to add an image grid?","cells":[{"id":0,"value":"# Wikification Part II: How to add an image grid?","pinned":false,"hidden":false,"mode":"md"},{"id":460,"value":"The [Part I](https://observablehq.com/@winkjs/how-to-build-a-naive-wikification-tool) of this post described a naive entity linking approach using [WinkNLP’s](https://github.com/winkjs/wink-nlp) [custom entities](https://winkjs.org/wink-nlp/custom-entities.html) and Wikipedia's [search API](https://www.mediawiki.org/wiki/API:Search). In addition to highlighting the concepts and linking to the relevant Wikipedia pages, the Part II also fetches the images from the Wikipedia pages and display them neatly below the Wikified text.\n\nLike the [Part I](https://observablehq.com/@winkjs/how-to-build-a-naive-wikification-tool), the searches are online with a “delay” injected after each search to respect rate limits. Each image fetch adds a bit extra time to the overall processing. Thus the **code may take several seconds to complete** the wikification process. Total time taken will depend on the number of searches required i.e. number of potential entities detected and the number of images to be fetched.","pinned":false,"hidden":false,"mode":"md"},{"id":505,"value":"### Example Text","pinned":false,"hidden":false,"mode":"md"},{"id":511,"value":"text = `Apollo 11 was the spaceflight that first landed humans on the Moon. \nCommander Neil Armstrong and lunar module pilot Buzz Aldrin formed the \nAmerican crew that landed the Apollo Lunar Module Eagle. The Lunar Module \nor LM-5 Eagle was manufactured by Grumman Corporation, which later merged\nwith Northrop Corporation in 1995 and became Northrop Grumman Corporation.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":517,"value":"### Wikified Text","pinned":false,"hidden":false,"mode":"md"},{"id":228,"value":"{\n const doc = nlp.readDoc( text );\n const potentialEntities = doc.customEntities().out();\n const results = (await searchAllTerms( potentialEntities )); // return results;\n let images = '';\n \n for ( let i = 0; i < potentialEntities.length; i += 1 ) {\n const e = doc.customEntities().itemAt( i );\n const match = findBestMatch( results[ i ], e );\n if ( match !== null ) { \n e.markup(``, '');\n const image = (await getImage( match.title )).query.pages[match.pageid].thumbnail;\n images += ( image ) ? `

`: '';\n \n }\n }\n return html `

${doc.out(its.markedUpText)}

${images}

`;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":601,"value":"This is a very simple implementation in ~30 source code lines. A full scale implementation will need to use advance techniques such as co-reference resolution, disambiguation and ranking the search results.","pinned":false,"hidden":false,"mode":"md"},{"id":468,"value":"---","pinned":false,"hidden":false,"mode":"md"},{"id":189,"value":"### Setup","pinned":false,"hidden":false,"mode":"md"},{"id":194,"value":"winkNLP = (await import('https://cdn.skypack.dev/wink-nlp')).default;","pinned":false,"hidden":false,"mode":"ojs"},{"id":195,"value":"model = (await import('https://cdn.skypack.dev/wink-eng-lite-web-model')).default;","pinned":false,"hidden":false,"mode":"ojs"},{"id":199,"value":"nlp = winkNLP( model );","pinned":false,"hidden":false,"mode":"ojs"},{"id":202,"value":"its = nlp.its;","pinned":false,"hidden":false,"mode":"ojs"},{"id":205,"value":"patterns = [\n {\n name: 'nounPhrase',\n patterns: [ '[PROPN] [|PROPN] [|PROPN] [|PROPN]' ]\n },\n {\n name: 'nounPhrase',\n patterns: [ '[PROPN] [ADJ|PROPN] [|PROPN] [|PROPN]' ]\n },\n {\n name: 'nounPhrase',\n patterns: [ '[PROPN|ADJ] [PROPN]' ]\n },\n {\n name: 'nounPhrase',\n patterns: [ '[PROPN] [CARDINAL]' ]\n },\n {\n name: 'simpleADJ',\n patterns: [ '[ADJ]' ]\n }\n];","pinned":false,"hidden":false,"mode":"ojs"},{"id":213,"value":"patternCount = nlp.learnCustomEntities( patterns, { matchValue: false, useEntity: true, usePOS: true } );","pinned":false,"hidden":false,"mode":"ojs"},{"id":541,"value":"findBestMatch= function ( ri, e ) {\n let index = null;\n if ( ri.query.searchinfo.totalhits === 0 ) return index;\n if ( e.out( its.type ) === 'simpleADJ' && !e.out( its.value ).match( /^[A-Z]/ ) ) return index;\n // Extract normalized text of custom entity and split on spaces and rejoin on vertical bar to form regex.\n const rgx = RegExp( e.out(its.normal).split( /\\s+/ ).join( '|' ), 'i' );\n for ( let k = 0; k < ri.query.search.length; k += 1 ) {\n const riqsk = ri.query.search[ k ];\n if ( riqsk.title.match(rgx) || riqsk.snippet.match(rgx) )\n return { pageid: riqsk.pageid, title: riqsk.title }\n }\n return null;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":641,"value":"getImage = async function ( term ) {\n const url = `https://en.wikipedia.org/w/api.php?format=json&origin=*&action=query&prop=pageimages&pithumbsize=200&titles=${term}`\n const response = await fetch(url) // d3.json\n if (!response.ok) {\n throw response.statusText;\n }\n \n return response.json();\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":3,"value":"searchAllTerms = async function ( terms ) {\n const allResults = [];\n for ( var i = 0; i < terms.length; i += 1 ) {\n Promises.delay(30); // Be nice to Wikipedia\n const r = await search( terms[ i ] );\n allResults.push( r );\n }\n return allResults;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":46,"value":"search = async function ( term ) {\n const url = `https://en.wikipedia.org/w/api.php?format=json&origin=*&action=query&list=search&srsearch=${term}`\n const response = await fetch(url) // d3.json\n if (!response.ok) {\n throw response.statusText;\n }\n \n return response.json();\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":327,"value":"","pinned":false,"hidden":false,"mode":"html"},{"id":532,"value":"---\n### About\n[**WinkNLP**](https://winkjs.org/) is a **developer friendly** JavaScript library for Natural Language Processing (NLP). Designed specifically to make development of NLP solutions **easier** and **faster**, winkNLP is optimized for the right balance of performance and accuracy. The package can handle large amount of raw text at speeds **over 525,000 tokens/second** for the entire NLP pipeline. And with a **test coverage of ~100%**, [wink-nlp](https://winkjs.org/) is a tool for building production grade systems with confidence.","pinned":false,"hidden":false,"mode":"md"}],"files":"$W31","stdlib":"1","titling":"auto"}