31:T652,md`# wtf_wikipedia Tutorial [wtf_wikipedia](https://github.com/spencermountain/wtf_wikipedia) is reasonably good at parsing wikipedia's ad-hoc syntax, and turning it into JSON. (though still not perfect!) But in addition to parsing, a _good-amount_ of finesse is needed to get the data you want, from the the pages you want. Suppose we want to find the **birth-dates of all current NBA basketball players**, or the **set locations of all bollywood movies**. Wikipedia has it all, we just have a few things to do to make this happen ### 1) API or data-dump? if you're fetching 1-2 thousand articles, you can do safely do so from the wikipedia api, without malice. It is critical to do this serially, over some time, instead of all-at-once. it is courteous also to pass a \`Api-User-Agent\`, to the sys-ops. It is also a good idea to bunch a few articles per request, to minimize http load. You can see a [good example of this here](https://github.com/spencermountain/wtf_wikipedia/blob/master/examples/02-fetch-list.js). but, briefly: \`\`\`js wtf.fetch(['Royal Cinema', 'Aldous Huxley'], 'en', { 'Api-User-Agent': 'spencermountain@gmail.com' }).then((docList) => { let infoboxes = docList.map(doc => { return { title: doc.title(), infobox: doc.infobox() } }); console.log(infoboxes); }); \`\`\` --- if you need more data than that, your best bet is to process a [wikipedia dump](https://dumps.wikimedia.org/). Using the [dumpster-dive](https://github.com/spencermountain/dumpster-dive/) library, you can load a custom-parsed english wikipedia into mongodb with relative ease. `32:T6eb,md` ### 2) Finding the data if we want to get the current roster of the Toronto Raptors, we can go to page [Toronto Raptors](https://en.wikipedia.org/wiki/Toronto_Raptors), but we'll find that the information is not on that page at all. Current sports rosters are (usually) on a specific-season's page, this being [2018–19_Toronto_Raptors_season](https://en.wikipedia.org/wiki/2018%E2%80%9319_Toronto_Raptors_season), (usually) under the section **'Roster'**: Most structured data in wikipedia comes from Templates, Tables, or Lists. ### Templates Templates come in mostly ad-hoc, unstandardised data formats, like ['Template:HorseDeathYear'](https://en.wikipedia.org/wiki/Template:HorseDeathYear) or ['Election_box_candidate_with_party_link_coalition_1918'](https://en.wikipedia.org/wiki/Template:Election_box_candidate_with_party_link_coalition_1918). There are [634,755](https://s3-us-west-1.amazonaws.com/spencer-scratch/allTemplates-2018-10-26.tsv) templates in wikipedia, as of Nov 2018, many with their own unique syntax. A fair percent are undocumented, and many are largely in-use contrary to specification or design. There is template merging, renaming, and deleting. 😅 Some templates are called an **'Infobox'**, which have a mostly-standard way of being rendered on the page. These are seperated in wtf_wikipedia output. You can get them using \`.infoboxes()\` Some templates **~transclude~** data, so their information is pulled from elsewhere. This information will not be available when you parse the page with wtf_wikipedia. A good example is the list on [Template:Cartier_Horse_of_the_Year](https://en.wikipedia.org/w/index.php?title=Template:Cartier_Horse_of_the_Year). To get this information, you'll have to parse the template itself: `33:[] 2f:{"title":"wtf_wikipedia Tutorial","cells":[{"id":0,"value":"$31","pinned":false,"hidden":false,"mode":"ojs"},{"id":6,"value":"$32","pinned":false,"hidden":false,"mode":"ojs"},{"id":111,"value":"wtf.fetch('Template:Cartier_Horse_of_the_Year').then(doc => doc.template('navbox'))","pinned":true,"hidden":false,"mode":"ojs"},{"id":121,"value":"md`\n\n---\n\nAdditionaly, some information like multilingual **interwiki-links** are pulled-in dynamically from [wikidata](https://www.wikidata.org/wiki/Wikidata:Main_Page).\n\n\n### Getting the 'Toronto Raptors' roster:\nAs I said, sometimes you have to look-around a little bit for data. Suppose we want the current list of players on a basketball team. This information is in wikipedia, you just have to look-around a little bit.\n\nThis is a good example:\n\nFor one, the **Toronto_Raptors** page doesn't show their current roster. It's hidden over on **[2018–19_Toronto_Raptors_season](https://en.wikipedia.org/wiki/2018%E2%80%9319_Toronto_Raptors_season)**:\n![image](https://user-images.githubusercontent.com/399657/49041865-68c51380-f194-11e8-9568-77637bbba1a7.png)\n\nlet's try to get it:\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":20,"value":"wtf.fetch('2018–19_Toronto_Raptors_season').then(doc=> doc.section('Roster').json())","pinned":true,"hidden":false,"mode":"ojs"},{"id":44,"value":"md`if you poke around, can see we have some template data in this Section, under \\'templates\\'.\n\nThe template is called \\'Toronto Raptors roster\\' - ***Urgh***, but it looks like this roster information is ***transcluded***! Rats. \n\nWe can confirm this by looking at the wikiscript on the page itself:\n![image](https://user-images.githubusercontent.com/399657/49029548-e0824680-f172-11e8-9aa3-104d124a577e.png)\n\nLooks like the page we want is [Template:Toronto Raptors roster](https://en.wikipedia.org/wiki/Template:Toronto_Raptors_roster).\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":50,"value":"wtf.fetch('Template:Toronto Raptors roster').then(doc=>doc.json())","pinned":true,"hidden":false,"mode":"ojs"},{"id":55,"value":"md`Take a look at the data, you can see this page has 2 sections, and information about both. The data we want looks like it's in the first section, and is using **Template:Player2**.\n\nWe can grab them all, using a simple ***doc.templates().filter()***, or the even, with **doc.templates('player2')**:\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":62,"value":"wtf.fetch('Template:Toronto Raptors roster').then(doc => {\n let data = doc.templates('player2')\n return data.map(t => {\n let obj = t.json()\n return `* ${obj.first || ''} ${obj.last || ''}`\n }).join('\\n')\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":75,"value":"md`if we want to do the same for another basketball team, we can get a list of teams, and wrap this into a function. Some teams may have different formats. This is a constant, but sovlable problem, using defensive and adaptive code.\n\nIn fact, let's try to get the list of NBA teams, now that we're here.\n\n### Tables\nA lot of great data lives in HTML-based tables in wikipedia.\n![image](https://user-images.githubusercontent.com/399657/49040776-77f69200-f191-11e8-9be8-fa6d1b4ef4bc.png)\n\nHere, we'll grab this table of teams on the [NBA page](https://en.wikipedia.org/wiki/National_Basketball_Association#Teams):\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":81,"value":"wtf.fetch('National_Basketball_Association').then(doc=>{\n let table= doc.section('teams').tables()[0]\n //grab just the text-information\n let rows= table.keyValue()\n //rows with team names\n rows=rows.filter(r => r.Team)\n //render it as we like\n return rows.map(r=> r.Team +', '+r.Capacity).join('\\n')\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":88,"value":"md`### Conclusion\nI know this is pretty tricky, and verbose. There are many times I've wanted to abstract things higher, so that users of ***wtf_wikipedia*** don't need to know any weird wikipedia internals. \n\nFor now, this library attempts to parse syntax only, and make minimal determinations, or decisions. Wikipedia is a huge API-surface to maintain a library on top of.\n\nLuckily, several people have written high-abstraction interfaces, like [wtf-mlb](https://github.com/spencermountain/wtf-mlb), which runs-around this tricksy wikipedia format stuff.\n\nIf you get-around to making one, please [let me know](http://twitter.com/spencermountain)\n\nThank you\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":60,"value":"","pinned":false,"hidden":false,"mode":"ojs"},{"id":11,"value":"html`\n

API

\n\n full docs are on github here\n\n\n

version

\nthis page is running the latest version, \n ${wtf.version}.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":14,"value":"style = html``","pinned":false,"hidden":false,"mode":"ojs"},{"id":7,"value":"wtf=require('wtf_wikipedia')","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W33","stdlib":"1","titling":"auto"}