31:[] 2f:{"title":"How to build a naive wikification tool?","cells":[{"id":0,"value":"# How to build a naive wikification tool?","pinned":false,"hidden":false,"mode":"md"},{"id":460,"value":"Wikification or **entity linking** is a process of linking entities such as famous persons, locations or objects to the relevant Wikipedia pages. This tool identifies potential entities via the [custom entities](https://winkjs.org/wink-nlp/custom-entities.html) feature of [winkNLP](https://github.com/winkjs/wink-nlp). It searches the potential entries in Wikipedia using its [search API](https://www.mediawiki.org/wiki/API:Search). Afterwards, it scans each set of result for the best match. If a match is found, tool links the entity to the matched Wikipedia page using the [markup()](https://winkjs.org/wink-nlp/visualizing-markup.html) method of [winkNLP](https://github.com/winkjs/wink-nlp). \n\nWikipedia searches are online. There is an added delay after each search to respect rate limits. Thus the code may take several seconds to complete the wikification process. Total time taken will depend on the number of searches required i.e. number of potential entities detected.","pinned":false,"hidden":false,"mode":"md"},{"id":505,"value":"### Example Text","pinned":false,"hidden":false,"mode":"md"},{"id":511,"value":"text = `Apollo 11 was the spaceflight that first landed humans on the Moon. \nCommander Neil Armstrong and lunar module pilot Buzz Aldrin formed the \nAmerican crew that landed the Apollo Lunar Module Eagle. The Lunar Module \nor LM-5 Eagle was manufactured by Grumman Corporation, which later merged\nwith Northrop Corporation in 1995 and became Northrop Grumman Corporation.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":517,"value":"### Wikified Text","pinned":false,"hidden":false,"mode":"md"},{"id":228,"value":"{\n const wikiBaseUrl = 'https://en.wikipedia.org/wiki?';\n const doc = nlp.readDoc( text );\n const potentialEntities = doc.customEntities().out();\n const results = (await searchAllTerms( potentialEntities )); // return results;\n doc.customEntities().each( (e, i) => {\n const match = findBestMatch( results[ i ], e );\n if ( match !== null ) \n e.markup(``, '');\n } );\n return html `

${doc.out(its.markedUpText)}

`;\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":601,"value":"This is a very simple implementation in ~30 source code lines. A full scale implementation will need to use advance techniques such as co-reference resolution, disambiguation and ranking the search results.","pinned":false,"hidden":false,"mode":"md"},{"id":468,"value":"---","pinned":false,"hidden":false,"mode":"md"},{"id":189,"value":"### Setup","pinned":false,"hidden":false,"mode":"md"},{"id":194,"value":"winkNLP = (await import('https://cdn.skypack.dev/wink-nlp')).default;","pinned":false,"hidden":false,"mode":"ojs"},{"id":195,"value":"model = (await import('https://cdn.skypack.dev/wink-eng-lite-web-model')).default;","pinned":false,"hidden":false,"mode":"ojs"},{"id":199,"value":"nlp = winkNLP( model );","pinned":false,"hidden":false,"mode":"ojs"},{"id":202,"value":"its = nlp.its;","pinned":false,"hidden":false,"mode":"ojs"},{"id":205,"value":"patterns = [\n {\n name: 'nounPhrase',\n patterns: [ '[PROPN] [|PROPN] [|PROPN] [|PROPN]' ]\n },\n {\n name: 'nounPhrase',\n patterns: [ '[PROPN] [ADJ|PROPN] [|PROPN] [|PROPN]' ]\n },\n {\n name: 'nounPhrase',\n patterns: [ '[PROPN|ADJ] [PROPN]' ]\n },\n {\n name: 'nounPhrase',\n patterns: [ '[PROPN] [CARDINAL]' ]\n },\n {\n name: 'simpleADJ',\n patterns: [ '[ADJ]' ]\n }\n];","pinned":false,"hidden":false,"mode":"ojs"},{"id":213,"value":"patternCount = nlp.learnCustomEntities( patterns, { matchValue: false, useEntity: true, usePOS: true } );","pinned":false,"hidden":false,"mode":"ojs"},{"id":541,"value":"findBestMatch= function ( ri, e ) {\n let index = null;\n if ( ri.query.searchinfo.totalhits === 0 ) return index;\n if ( e.out( its.type ) === 'simpleADJ' && !e.out( its.value ).match( /^[A-Z]/ ) ) return index;\n // Extract normalized text of custom entity and split on spaces and rejoin on vertical bar to form regex.\n const rgx = RegExp( e.out(its.normal).split( /\\s+/ ).join( '|' ), 'i' );\n for ( let k = 0; k < ri.query.search.length; k += 1 ) {\n const riqsk = ri.query.search[ k ];\n if ( riqsk.title.match(rgx) || riqsk.snippet.match(rgx) )\n return { pageid: riqsk.pageid, title: riqsk.title }\n }\n return null;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":3,"value":"searchAllTerms = async function ( terms ) {\n const allResults = [];\n for ( var i = 0; i < terms.length; i += 1 ) {\n Promises.delay(30); // Be nice to Wikipedia\n const r = await search( terms[ i ] );\n allResults.push( r );\n }\n return allResults;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":46,"value":"search = async function ( term ) {\n const url = `https://en.wikipedia.org/w/api.php?format=json&origin=*&action=query&list=search&srsearch=${term}`\n const response = await fetch(url) // d3.json\n if (!response.ok) {\n throw response.statusText;\n }\n \n return response.json();\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":327,"value":"","pinned":false,"hidden":false,"mode":"html"},{"id":532,"value":"---\n### About\n[**WinkNLP**](https://winkjs.org/) is a **developer friendly** JavaScript library for Natural Language Processing (NLP). Designed specifically to make development of NLP solutions **easier** and **faster**, winkNLP is optimized for the right balance of performance and accuracy. The package can handle large amount of raw text at speeds **over 525,000 tokens/second** for the entire NLP pipeline. And with a **test coverage of ~100%**, [wink-nlp](https://winkjs.org/) is a tool for building production grade systems with confidence.","pinned":false,"hidden":false,"mode":"md"}],"files":"$W31","stdlib":"1","titling":"auto"}