31:[] 2f:{"title":"How to do stemming and lemmatization?","cells":[{"id":0,"value":"# How to do stemming and lemmatization?","pinned":false,"hidden":false,"mode":"md"},{"id":5,"value":"[WinkNLP](https://github.com/winkjs/wink-nlp) has a rich set of features for text pre-processing or normalization. These include a stemmer and a lemmatizer. The stemmer is based on [Porter Stemmer algorithm version 2.0](https://snowballstem.org/algorithms/english/stemmer.html). The lemmatizer is inspired by [Wordnet's](https://wordnet.princeton.edu/) lemmatizer. As illustrated in the following example, both the operations can be easily accomplished using [`out()`]() method with [`its`]() helpers:","pinned":false,"hidden":false,"mode":"md"},{"id":18,"value":"{\n const doc = nlp.readDoc( text );\n let display = init();\n \n display += startRow( '.out()' );\n doc.tokens().each( ( t ) => display += content( t, 'value' ) );\n display += endRow();\n \n display += startRow( '.out(its.lemma)' );\n doc.tokens().each( ( t ) => display += content( t, 'lemma' ) );\n display += endRow();\n \n display += startRow( '.out(its.stem)' );\n doc.tokens().each( ( t ) => display += content( t, 'stem' ) );\n display += endRow();\n \n display += end();\n return html`${display}`;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":73,"value":"There are interesting observations about lemma and stem in the above example:\n

Lemmatization returned have as the original form of had, whereas stemmer failed to do so.\n
Stemmer stemmed beautiful to beauti, which is not a valid English word; same thing happened to the word voice.\n
Stemmer converted the singing to sing, whereas lemmatizer did not perform any conversion since it is a noun and not a verb. \n

","pinned":false,"hidden":false,"mode":"html"},{"id":65,"value":"Lemmatization requires morphological analysis of a word in the given context i.e. its part-of-speech. Stemming is a naive way to lemmatize — it does not consider the context of the word and algorithmically replaces affixes. A typical example is to drop the \"-ing\" or \"-ed\" suffix to derive the stem. It may not always yield a proper dictionary word.","pinned":false,"hidden":false,"mode":"md"},{"id":68,"value":"The choice between lemmatization or stemming depends on the context. Consider an application that is using word vectors. Here stemming is not a good choice as it sometimes yields an invalid word, which will not have a vector. With lemmatization this situation will never arise. Performance could be another consideration. Stemming speed is generally faster as it does not depend on part-of-speech. For many applications stemming is good enough — typical example is TF-IDF or BM25 text search.","pinned":false,"hidden":false,"mode":"md"},{"id":23,"value":"md`---\n### Setup`","pinned":false,"hidden":false,"mode":"ojs"},{"id":26,"value":"winkNLP = (await import('https://cdn.skypack.dev/wink-nlp')).default;","pinned":false,"hidden":false,"mode":"ojs"},{"id":29,"value":"model = (await import('https://cdn.skypack.dev/wink-eng-lite-web-model')).default;","pinned":false,"hidden":false,"mode":"ojs"},{"id":32,"value":"nlp = winkNLP( model );","pinned":false,"hidden":false,"mode":"ojs"},{"id":35,"value":"its = nlp.its;","pinned":false,"hidden":false,"mode":"ojs"},{"id":81,"value":"text = 'My mother had a beautiful singing voice.';","pinned":false,"hidden":false,"mode":"ojs"},{"id":86,"value":"cssClass = {\n let cssc = Object.create( null );\n cssc.had = 'had';\n cssc.beautiful = 'beautiful';\n cssc.voice = 'beautiful';\n cssc.singing = 'singing';\n return cssc;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":129,"value":"init = function () {\n return `

Example sentence

${text}

`;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":132,"value":"startRow = function ( h ) {\n return ``;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":102,"value":"content = function ( t, itsType ) {\n return ``;;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":140,"value":"endRow = function () {\n return '';\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":146,"value":"end = function () {\n return '

${h}	${t.out( its[itsType] )}

';\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":213,"value":"md`---\n### CSS`","pinned":false,"hidden":false,"mode":"ojs"},{"id":159,"value":"html`\n `","pinned":false,"hidden":false,"mode":"ojs"},{"id":20,"value":"md`---\n### About\n[**WinkNLP**](https://winkjs.org/) is a **developer friendly** JavaScript library for Natural Language Processing (NLP). Designed specifically to make development of NLP solutions **easier** and **faster**, winkNLP is optimized for the right balance of performance and accuracy. The package can handle large amount of raw text at speeds **over 525,000 tokens/second** for the entire NLP pipeline. And with a **test coverage of ~100%**, [wink-nlp](https://winkjs.org/) is a tool for building production grade systems with confidence.`","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W31","stdlib":"1","titling":"auto"}