30:[] 2e:{"title":"PSET 3","cells":[{"id":0,"value":"md`# PSET 3`","pinned":false,"hidden":false,"mode":"ojs"},{"id":10,"value":"md`## Data Management\nThis problem set is meant to create a foundation of data management using Zebras library and other Javascript manipulation. We emphasize code readability and reusability over performance. Code performance can be measured in many different ways, and is by no means the focus of this course, however, we expect your code to be non-repetitive, clean, and efficient enough to deal with medium-to-large datasets.\n\nYou can always look up [Zebras documentation](https://zebrasjs.com/) for help.\n\n## Submission Instructions\nSubmit the link of your assignment on Stellar. \\n\n**Remember to publish your notebook so it is publicly accesible!** \\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":19,"value":"md`## 1. Loading data\nLoad [\\`MIG_18022019163824069.csv\\`](https://gist.githubusercontent.com/cesandoval/b834ac93c07e03ec5205843b97f68017/raw/6dff0d45ed26352a7a3c7afb4ace0bad1ce8ba20/MIG_18022019163824069.csv)`\n","pinned":false,"hidden":false,"mode":"ojs"},{"id":183,"value":"// Loading necessary library here\nz = require('https://bundle.run/zebras@0.0.11')","pinned":true,"hidden":false,"mode":"ojs"},{"id":247,"value":"d3 = require(\"d3\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":243,"value":"Url='https://gist.githubusercontent.com/cesandoval/b834ac93c07e03ec5205843b97f68017/raw/6dff0d45ed26352a7a3c7afb4ace0bad1ce8ba20/MIG_18022019163824069.csv'","pinned":true,"hidden":false,"mode":"ojs"},{"id":246,"value":"file=d3.csv(Url)","pinned":true,"hidden":false,"mode":"ojs"},{"id":45,"value":"md`\n## 2. Unique value\n\nTo start us off, we need to determine how many different countries of birth we have registered in our dataset. Let's take a look at unique countries of birth in the data set.\n- Get the unique values for *Country of Birth*.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":26,"value":"// Your code here\nCountryBirth=z.unique(z.getCol('Country of birth/nationality', file))","pinned":true,"hidden":false,"mode":"ojs"},{"id":57,"value":"md`\n## 3. Grouping and sorting data\n\nThere are a lot of countries represented in our dataset! Let's explore which countries have the highest number of \\`Inflows of foreign population by nationality, Outflows of foreign population by nationality, Stock of foreign-born population by country of birth\\`.\n- Group your data and find the total number of \\`Inflows of foreign population by nationality, Outflows of foreign population by nationality, Stock of foreign-born population by country of birth\\` **per country**, and sum them in a Zebras table.\n- Call your new data frame \\`df_summed\\`.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":338,"value":"SliceFile= z.filter(r=>r.Variable=='Inflows of foreign population by nationality'||r.Variable=='Outflows of foreign population by nationality'|| r.Variable=='Stock of foreign-born population by country of birth', file)","pinned":true,"hidden":false,"mode":"ojs"},{"id":299,"value":"Group = {\n var fileParse=z.parseNums([\"Value\"], SliceFile)\n var m=d3.nest()\n .key(function(d) { return d['Country of birth/nationality']; })\n .key(function(d) { return d['Variable']; })\n .rollup(function(v) { return {\n total: parseInt(d3.sum(v, function(d) { return d.Value; }))}})\n .object(fileParse)\n return m\n}\n","pinned":true,"hidden":false,"mode":"ojs"},{"id":488,"value":"\narr = Object.keys(Group).map(function (key) { return Group[key]; })","pinned":true,"hidden":false,"mode":"ojs"},{"id":517,"value":"df_summed={\n let frame=[]\n for (var i=0; ix['Country of birth/nationality'],SliceFile))\n","pinned":true,"hidden":false,"mode":"ojs"},{"id":387,"value":"\nresult=Group2.map(function (e) {\n var element = {};\n element[e.group] = e.count; \n return element;\n})\n ","pinned":true,"hidden":false,"mode":"ojs"},{"id":425,"value":"","pinned":false,"hidden":false,"mode":"ojs"},{"id":114,"value":"md`## 4. Joining data\nFor this question, we want to compare how the immigration stats differ across continents. You can find a list of Countries, County Codes, and Continents [here](https://gist.githubusercontent.com/cesandoval/b834ac93c07e03ec5205843b97f68017/raw/0c14fa1b1f65a610491417b2894fceecde5f4260/country-and-continent-codes-list-csv_csv.csv)\n\n- Load this data.\n- Create a data frame that only keeps the \\`Country Codes \\` and the \\`Continent\\`.\n- Join your datasets by \\`Country Code\\`\n- Create a new aggregated dataset that counts the number of \\`Inflows of foreign population by nationality, Outflows of foreign population by nationality, Stock of foreign-born population by country of birth\\` by \\`Continent\\`\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":91,"value":"// Your code here\nUrl2=`https://gist.githubusercontent.com/cesandoval/b834ac93c07e03ec5205843b97f68017/raw/0c14fa1b1f65a610491417b2894fceecde5f4260/country-and-continent-codes-list-csv_csv.csv`","pinned":false,"hidden":false,"mode":"ojs"},{"id":92,"value":"// 1 Load this data\nfile2=d3.csv(Url2)","pinned":true,"hidden":false,"mode":"ojs"},{"id":112,"value":"// 2 Create a data frame that only keeps the Country Codes and the Continent.\nNewFile2=z.pickCols(['Continent_Name','Three_Letter_Country_Code'],file2)","pinned":true,"hidden":false,"mode":"ojs"},{"id":451,"value":"//Join your datasets by Country Code (a complicated code but the result should be correct)\nmigration_join = SliceFile.map(x => Object.assign(x, NewFile2.find(y => y['Three_Letter_Country_Code'] == x['CO2'])));","pinned":true,"hidden":false,"mode":"ojs"},{"id":458,"value":"//Create a new aggregated dataset (please scroll down to see the table result)\nmigration_join_group = {\n var fileParse=z.parseNums([\"Value\"], SliceFile)\n var m=d3.nest()\n.key(function(d){return d['Continent_Name']})\n.key(function(d){return d['Variable']})\n.rollup(function(v){return {\n total: parseInt(d3.sum(v, function(d) { return d.Value; }))}})\n.object(migration_join)\n return m\n}\n","pinned":true,"hidden":false,"mode":"ojs"},{"id":618,"value":"arr2 = Object.keys(migration_join_group).map(function (key) { return migration_join_group[key]; })","pinned":true,"hidden":false,"mode":"ojs"},{"id":644,"value":"arr2_clip=arr2.slice(0,arr2.length-1)","pinned":false,"hidden":false,"mode":"ojs"},{"id":623,"value":"Continents=z.unique(z.getCol('Continent_Name', migration_join))","pinned":true,"hidden":false,"mode":"ojs"},{"id":631,"value":"Continents_clip=Continents.slice(0, Continents.length-1)","pinned":false,"hidden":false,"mode":"ojs"},{"id":628,"value":"df_summed_cont={\n let frame=[]\n for (var i=0; i 100,000\n - \\`Medium Immigration\\` > 50,000 & < 100,000\n - \\`Low Immigration\\` < 50,000\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":723,"value":"z.unique(z.getCol('Variable',file))","pinned":true,"hidden":false,"mode":"ojs"},{"id":720,"value":"SumAll={\n var fileParse=z.parseNums([\"Value\"], file)\n var m=z.unique(z.getCol('Variable',file))\n var IniFrame=[]\n for (var i=0; ir.Variable== m[a], fileParse)\n var CountrySpeSum=z.gbSum(\"Value\", z.groupBy(r => r['Country of birth/nationality'], FlowType))\n //for each flow type, create a new dataframe\n var subframe=[]\n for (var i=0; ix.group==CountryBirth[i],CountrySpeSum)\n subframe.push({\"Country of birth\":CountryBirth[i],\"Type\":parseInt(CountryOne[0][\"sum\"])})\n }\n IniFrame= z.addCol(m[a], z.getCol(\"Type\",subframe), IniFrame) \n } return IniFrame\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":747,"value":"z.print(SumAll)","pinned":false,"hidden":false,"mode":"ojs"},{"id":95,"value":"// Your code here\nSumStock={\n var fileParse=z.parseNums([\"Value\"], file)\n var Stock16=z.filter(r =>r.VAR== \"B14\" && r.YEA == \"2016\", fileParse)\n var StockSum=z.gbSum(\"Value\", z.groupBy(r => r['Country of birth/nationality'], Stock16))\n var StockSumFrame=[]\n for (var i=0; ix.group==CountryBirth[i],StockSum)\n StockSumFrame.push({\"Country of birth\":CountryBirth[i],\"2016 Stock of foreign-born population\": CountrySum[0][\"sum\"]})\n }\n return StockSumFrame\n}\n ","pinned":true,"hidden":false,"mode":"ojs"},{"id":664,"value":"z.print(SumStock)","pinned":false,"hidden":false,"mode":"ojs"},{"id":128,"value":"// Your code here\nHighImg={\n var fileParse=z.parseNums([\"Value\"], file)\n var Stock16=z.filter(r =>r.VAR== \"B14\" && r.YEA == \"2016\", fileParse)\n var StockSum=z.gbSum(\"Value\", z.groupBy(r => r['Country of birth/nationality'], Stock16))\n var StockSumFrame=[]\n for (var i=0; ix.group==CountryBirth[i],StockSum)\n if (CountrySum[0][\"sum\"]> 100000){\n StockSumFrame.push({\"Country of birth: High Immigration\":CountryBirth[i],\"2016 Stock of foreign-born population\": CountrySum[0][\"sum\"]})\n }}\n return StockSumFrame\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":698,"value":"MedImg={\n var fileParse=z.parseNums([\"Value\"], file)\n var Stock16=z.filter(r =>r.VAR== \"B14\" && r.YEA == \"2016\", fileParse)\n var StockSum=z.gbSum(\"Value\", z.groupBy(r => r['Country of birth/nationality'], Stock16))\n var StockSumFrame=[]\n for (var i=0; ix.group==CountryBirth[i],StockSum)\n if (CountrySum[0][\"sum\"]> 50000 && CountrySum[0][\"sum\"] < 100000){\n StockSumFrame.push({\"Country of birth: Medium Immigration\":CountryBirth[i],\"2016 Stock of foreign-born population\": CountrySum[0][\"sum\"]})\n }}\n return StockSumFrame\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":701,"value":"LowImg={\n var fileParse=z.parseNums([\"Value\"], file)\n var Stock16=z.filter(r =>r.VAR== \"B14\" && r.YEA == \"2016\", fileParse)\n var StockSum=z.gbSum(\"Value\", z.groupBy(r => r['Country of birth/nationality'], Stock16))\n var StockSumFrame=[]\n for (var i=0; ix.group==CountryBirth[i],StockSum)\n if (CountrySum[0][\"sum\"]< 50000){\n StockSumFrame.push({\"Country of birth: Low Immigration\":CountryBirth[i],\"2016 Stock of foreign-born population\": CountrySum[0][\"sum\"]})\n }}\n return StockSumFrame\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":676,"value":"z.print(HighImg)","pinned":true,"hidden":false,"mode":"ojs"},{"id":704,"value":"z.print(MedImg)","pinned":true,"hidden":false,"mode":"ojs"},{"id":707,"value":"z.print(LowImg)","pinned":true,"hidden":false,"mode":"ojs"},{"id":151,"value":"md `## 6. Comparing data\n\nFor this question, we ask you to compare the inflow and outflow of population by nationality.\n\n- Create a data frame that has a column called \\`inflow-outflow\\`, which subtracts the aggregated number of \\`Inflows of foreign population by nationality\\`** - **\\`Outflows of foreign population by nationality\\` by \\`Country of birth/nationality\\`.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":171,"value":"// Your code here\ndf_netinflow={\n let frame=[]\n for (var i=0; i