32:[] 30:{"title":"Find Labor Hours To Generate OpenSouce NASA Code Since A Selected Date","cells":[{"id":0,"value":"md`# Find Labor Hours To Generate OpenSouce NASA Code Since A Selected Date`","pinned":false,"hidden":false,"mode":"ojs"},{"id":5,"value":"html `

This notebook was created as a way to quickly calculate approximately how much labor was spent developing NASA open-source code.

It leverages the fact that all code project informatin is stored in a public JSON.

There are sliders for end-users to explore the effect of light cleaning of the labor Hours data in code.json.

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":169,"value":"md `-----------------------------------------`","pinned":false,"hidden":false,"mode":"ojs"},{"id":15,"value":"html `

JSONs that contain information on open-source NASA code

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":31,"value":"codejson = d3.json('https://raw.githubusercontent.com/nasa/Open-Source-Catalog/master/code.json')","pinned":true,"hidden":false,"mode":"ojs"},{"id":825,"value":"html `NOTE: If you have a code.json from another federal agency, you should theoretically be able to substitute it into the variable above and see its results in the rest of this notebook.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":166,"value":"md `-----------------------------------------`","pinned":false,"hidden":false,"mode":"ojs"},{"id":119,"value":"md `## Filtering them to only certain projects by Licensing AND/OR Update Date`","pinned":false,"hidden":false,"mode":"ojs"},{"id":41,"value":"md `### Select Source Type`","pinned":false,"hidden":false,"mode":"ojs"},{"id":85,"value":"viewof source = radio({\n title: 'License Type',\n description: 'Code on code.json is both source types',\n options: [\n { label: 'open-source', value: 'openSource' },\n { label: 'government-source', value: 'governmentWideReuse' },\n { label: 'both', value: 'both'},\n ],\n value: 'openSource'\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":44,"value":"md `### Select Metadata Changed Date`","pinned":false,"hidden":false,"mode":"ojs"},{"id":62,"value":"viewof date_proj_added_after = date({\n // title: \"filter to projects added after date\", \n min: \"2000-01-01\",\n max: \"2060-12-31\",\n value: \"2016-01-01\",\n description: \"This filters code projects to only those after this date\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":162,"value":"md `-----------------------------------------`","pinned":false,"hidden":false,"mode":"ojs"},{"id":306,"value":"html `

Labor Hours Statistics

for selected code projects based on license type and date

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":244,"value":"labor_hours_all_selected_projects = sumLaborHoursOfSelected(codejson_filter_by_date_and_source,\"laborHours\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":344,"value":"html `From start of 2016 to end of this year, the average person year spend on open-source code development if a person is assumed to just work on this for 40 hours a week.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":260,"value":"avg_person_years_per_year = person_years_labor_all/years_assumed","pinned":false,"hidden":false,"mode":"ojs"},{"id":698,"value":"html `....that seems high...`","pinned":false,"hidden":false,"mode":"ojs"},{"id":701,"value":"html `Here's a quick chart that gives an indication of the wide range in reported labor hours numbers.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":625,"value":"html `Scroll with your mouse over the x axis to see all the repos`","pinned":false,"hidden":false,"mode":"ojs"},{"id":388,"value":"vl.markBar()\n .data(codejson_filter_by_date_and_source)\n .encode(\n vl.x().fieldN('name'),\n vl.y().fieldQ('laborHours'),\n vl.order().fieldQ('laborHours').sort(\"-x\")\n )\n .render()","pinned":false,"hidden":false,"mode":"ojs"},{"id":436,"value":"html `

These numbers have a lots of Uncertainty! Some are zero & 1.8 million hours for one repository seems unlikely to be true!

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":787,"value":"html `

Sources of Error in Labor Hours:

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":608,"value":"html `

Labor hours data come from one of two sources:

Code owner inputs an estimated labor hours when they fill out forms during the software release application.
If no estimated hours are given, an automated program is used that leverages some assumptions about how long it takes to write each line of code. Code.gov has information on this program.

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":612,"value":"html `

Reasons for error

Some code repositories have data in with the code. The data files sometimes get read by the program that calculates estimated labor hours, which greatly increases the estimated time required to have written the code.
- One example is the RT framework. It has unit tests that include data files that were likely counted as code
- Other higher than normal repositories are ontology related, which have a lot of data files designed in uncommon formats that were likely mistakenly read by the program as code.
Some code can't get read or is read poorly by the program that calculates estimated labor hours. Binar files are just one example. These projects might get an estimate of zero hours of labor.
Even when people enter the labor hours, they might guess poorly and frequently tend to low ball the number in our limited experience.

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":465,"value":"html `

Let's try to limit labor hours to reasonable ranges.

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":581,"value":"html `

Limit laborHours to a max value

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":453,"value":"viewof max_labor_hours = slider({\n min: 100,\n max: 75000,\n step: 100,\n value: 42300,\n format: \",\",\n description:\n \"Limit labor hours for any project to a maxinum of a certain amount:100 to 80,000 - in steps of 100\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":551,"value":"html `For context, the maximum laborHours per code project selected in the slider above is equivalent to one person working full time just on that for ${(max_labor_hours/40/52).toFixed(2)} years without vacation`","pinned":false,"hidden":false,"mode":"ojs"},{"id":576,"value":"html `If we assume, it is being worked by a team of 3 people, then ${(max_labor_hours/40/52/3).toFixed(2) } years.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":830,"value":"html `If we assume, it is being worked by a team of people for 1 year, then ${(max_labor_hours/40/52).toFixed(0) } people.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":586,"value":"html `

Limit laborHours to a min value

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":461,"value":"viewof min_labor_hours = slider({\n min: 100,\n max: 5000,\n step: 100,\n value: 600,\n format: \",\",\n description:\n \"Limit labor hours to a minimum of a certain amount:100 to 5,000 - in steps of 100\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":636,"value":"html `For context, the minimum laborHours selected in the slider above is equivalent to one person working full time just on that for ${min_labor_hours/40} weeks.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":693,"value":"html `If we assume, it is being worked by a team of 3 people, then ${(min_labor_hours/40/3).toFixed(0) } weeks.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":705,"value":"html `

Now let's find the labor hours on NASA open-source with those max and min limits applied:

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":492,"value":"laborHoursSummed_w_laborHoursMinMaxApplied = sumLaborHoursOfSelected(codejson_filter_by_date_and_source_minMaxLaborHoursApplied,\"laborHoursNew\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":681,"value":"html `If we limit projects to that min and max range of labor hours and sum all the hours since the selected date with the select source type, we get in people years per year. Or the average number of people working on open-source software as:`","pinned":false,"hidden":false,"mode":"ojs"},{"id":545,"value":"People_Year_OnAvg_since_2016_minMaxApplied = {return (convertedPeoplePerYear_minMaxApplied/years_assumed).toFixed(0)}","pinned":false,"hidden":false,"mode":"ojs"},{"id":709,"value":"html `Across all ${(codejson_filter_by_date_and_source.length)} open-source repositories published since the selected date of ${date_proj_added_after}, the current best estimate of labor spent is ${People_Year_OnAvg_since_2016_minMaxApplied} full-time people years per year. This assumes they only worked on open-source code and nothing else.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":730,"value":"viewof percent_of_time_people_spend_on_opensource_on_avg = slider({\n min: 0.00,\n max: 1.0,\n step: 0.01,\n value: 0.40,\n format: \",\",\n description:\n \"Percent of time each person spends on open-source code development on average.\"\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":723,"value":"html `In reality, most people probably only spend a fraction of their time on open-source code development. The other time being spent on closed-source development or other activities. If we assume on average people are spending ${percent_of_time_people_spend_on_opensource_on_avg*100}% of their time on one or more of these open-sourced projects, then ${(People_Year_OnAvg_since_2016_minMaxApplied/percent_of_time_people_spend_on_opensource_on_avg).toFixed(0)} people per year contribute to open-source code development at NASA.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":777,"value":"html `The estimate of 585 people contributing to open-source NASA code per year that appeared when you first opened the page (and before you changed any of the sliders) seems a more reasonale number than the total without any min or max labor hours per project limits though there is still undoubtedly a lot of uncertainty around even that number.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":364,"value":"md `__________`","pinned":false,"hidden":false,"mode":"ojs"},{"id":601,"value":"html `

Related Information

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":7,"value":"html `

Websites that contain NASA Open-Source Code

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":12,"value":"html`

The website that attempts to be the larger collection across the whole agency and only lists open-source is Code.NASA.Gov

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":25,"value":"html`

This JSON: https://raw.githubusercontent.com/nasa/Open-Source-Catalog/master/code.json is harvested by code.gov. It contains the NASA open-source (and government-source) code projects metadata that you can find on their site. We also harvest this JSON into code.nasa.gov where it is transformed slightly into catalog.json and as raw project data to build code.nasa.gov.

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":9,"value":"html`

There are a couple different places you can find lists of NASA open-source code.

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":13,"value":"html`

If you work in the federal government or inside NASA and would like to also see some closed or government source only code, you should check out software.nasa.gov as well

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":593,"value":"md `__________`","pinned":false,"hidden":false,"mode":"ojs"},{"id":596,"value":"md `__________`","pinned":false,"hidden":false,"mode":"ojs"},{"id":46,"value":"md `### Data Results From Just Date & Source Filter Selections`","pinned":false,"hidden":false,"mode":"ojs"},{"id":203,"value":"md `#### the code.json filtered by the source & date selections above:`","pinned":false,"hidden":false,"mode":"ojs"},{"id":205,"value":"codejson_filter_by_date_and_source","pinned":false,"hidden":false,"mode":"ojs"},{"id":109,"value":"md `#### intermediate results of code.json filtered by date and/or source below`","pinned":false,"hidden":false,"mode":"ojs"},{"id":71,"value":"codejson_array = getReleasesFromCodeJSON(codejson)","pinned":false,"hidden":false,"mode":"ojs"},{"id":98,"value":"codejson_filter_by_source_only = filterCodeJSONBySource(codejson_array,source)","pinned":false,"hidden":false,"mode":"ojs"},{"id":80,"value":"codejson_filter_by_date = filterCodeJSONByDate(codejson,date_proj_added_after)","pinned":false,"hidden":false,"mode":"ojs"},{"id":103,"value":"codejson_filter_by_date_and_source = filterCodeJSONBySource(codejson_filter_by_date,source)","pinned":false,"hidden":false,"mode":"ojs"},{"id":189,"value":"md `-----------------------------------------`","pinned":false,"hidden":false,"mode":"ojs"},{"id":54,"value":"md `## Functions Used`","pinned":false,"hidden":false,"mode":"ojs"},{"id":141,"value":"function filterCatalogByDate(catalog_array,date){\n return catalog_array.filter(catalog_array => catalog_array[\"Update_Date\"] > date);\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":134,"value":"function filterCatalogBySource(catalog_array){\n if (source == \"both\" || source == \"openSource\"){return catalog_array}\n else{\n return []\n }\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":55,"value":"function limitCatalogJsonByDate(catalog){\n var catalog_new = []\n for (var i=0; i< catalog.length; i++){\n // if(new Date(catalog[i][\"Update_Date\"]) <= new Date(date_proj_added_after)){\n if(catalog[i][\"Update_Date\"] >= date_proj_added_after){\n catalog_new.push(catalog[i])\n }\n }\n return catalog_new\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":68,"value":"function getReleasesFromCodeJSON(code_json){\n return code_json[\"releases\"]\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":75,"value":"function filterCodeJSONByDate(codejson,date){\n return codejson_array.filter(codejson_array => codejson_array[\"date\"][\"metadataLastUpdated\"] > date);\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":93,"value":"function filterCodeJSONBySource(code_json_array,source){\n if (source == \"both\"){return code_json_array}\n else{\n return code_json_array.filter(code_json_array => code_json_array[\"permissions\"][\"usageType\"] == source)\n }\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":233,"value":"function splitByWhetherLaborHoursZero(array_projects){\n let projects_w_zero_labor_hours = []\n let projects_w_some_labor_hours = []\n // test = array_projects.map(function(if (let proj)){\n // return product.id;\n // })\n projects_w_zero_labor_hours = array_projects.filter((item) => item.laborHours === 0)\n projects_w_some_labor_hours = array_projects.filter((item) => item.laborHours > 0)\n return {\" projects_w_zero_labor_hours\": projects_w_zero_labor_hours,\"projects_w_some_labor_hours\":projects_w_some_labor_hours}\n}\n","pinned":false,"hidden":false,"mode":"ojs"},{"id":235,"value":"projects_splitByWhetherLaborHoursZero = splitByWhetherLaborHoursZero(codejson_filter_by_date_and_source,\"laborHours\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":240,"value":"function sumLaborHoursOfSelected(array_projects,key){\n let sum_labor_hours = 0\n for (let x = 0; x < array_projects.length; x++){\n sum_labor_hours = array_projects[x][key] + sum_labor_hours\n }\n return sum_labor_hours\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":252,"value":"function convertHoursToWorkWeeks(labor_hours_all_selected_projects){\n return labor_hours_all_selected_projects/40\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":256,"value":"function convertHoursToPeopleYears(labor_hours_all_selected_projects){\n return labor_hours_all_selected_projects/40/52\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":475,"value":"function limit_laborHours_byMinMax(array_projects_filtered_date_source){\n let new_array = []\n for (let x = 0; x < array_projects_filtered_date_source.length; x++){\n let item = array_projects_filtered_date_source[x]\n if(item.laborHours < min_labor_hours){item.laborHoursNew = min_labor_hours}\n else if(item.laborHours > max_labor_hours){item.laborHoursNew = max_labor_hours}\n else{item.laborHoursNew = item.laborHours}\n \n new_array.push(item)\n }\n return new_array\n \n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":470,"value":"html `

Calculations below and functions above

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":248,"value":"codejson_filter_by_date_and_source","pinned":false,"hidden":false,"mode":"ojs"},{"id":258,"value":"person_years_labor_all = convertHoursToPeopleYears(labor_hours_all_selected_projects)","pinned":false,"hidden":false,"mode":"ojs"},{"id":265,"value":"current_date = new Date()","pinned":false,"hidden":false,"mode":"ojs"},{"id":274,"value":"function formatDate(date) {\n var d = new Date(date),\n month = '' + (d.getMonth() + 1),\n day = '' + d.getDate(),\n year = d.getFullYear();\n\n if (month.length < 2) \n month = '0' + month;\n if (day.length < 2) \n day = '0' + day;\n\n return [year, month, day].join('-');\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":275,"value":"current_date_1 = formatDate(current_date)","pinned":false,"hidden":false,"mode":"ojs"},{"id":299,"value":"current_year = parseInt(current_date_1.slice(0,4))+1","pinned":false,"hidden":false,"mode":"ojs"},{"id":300,"value":"since_year = parseInt(date_proj_added_after.slice(0,4))","pinned":false,"hidden":false,"mode":"ojs"},{"id":302,"value":"years_assumed = current_year-since_year","pinned":false,"hidden":false,"mode":"ojs"},{"id":262,"value":"avg_person_years_per_year","pinned":false,"hidden":false,"mode":"ojs"},{"id":291,"value":"html `

So does this mean `+avg_person_years_per_year+` people are writing open-source software per year if we assume they are just working on that and working 40 hours a week and 52 weeks a year? Seems high.... need to re-evaluate

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":326,"value":"date_proj_added_after","pinned":false,"hidden":false,"mode":"ojs"},{"id":321,"value":"typeof(date_proj_added_after)","pinned":false,"hidden":false,"mode":"ojs"},{"id":324,"value":"date_proj_added_after.slice(0,4)","pinned":false,"hidden":false,"mode":"ojs"},{"id":483,"value":"codejson_filter_by_date_and_source_minMaxLaborHoursApplied = limit_laborHours_byMinMax(codejson_filter_by_date_and_source_copy)","pinned":false,"hidden":false,"mode":"ojs"},{"id":520,"value":"codejson_filter_by_date_and_source_copy = JSON.parse(JSON.stringify(codejson_filter_by_date_and_source))","pinned":false,"hidden":false,"mode":"ojs"},{"id":557,"value":"max_labor_in_person_per_year = {return (max_labor_hours/40/52).toFixed(2)}","pinned":false,"hidden":false,"mode":"ojs"},{"id":538,"value":"convertedPeoplePerYear_minMaxApplied = convertHoursToPeopleYears(laborHoursSummed_w_laborHoursMinMaxApplied)","pinned":false,"hidden":false,"mode":"ojs"},{"id":638,"value":"min_labor_in_person_per_year = {return (min_labor_hours/40/52).toFixed(2)}","pinned":false,"hidden":false,"mode":"ojs"},{"id":379,"value":"md `_______`","pinned":false,"hidden":false,"mode":"ojs"},{"id":182,"value":"md `-----------------------------------------`","pinned":false,"hidden":false,"mode":"ojs"},{"id":35,"value":"md `## IMPORTS`","pinned":false,"hidden":false,"mode":"ojs"},{"id":384,"value":"import {vl} from '@vega/vega-lite-api'","pinned":false,"hidden":false,"mode":"ojs"},{"id":4,"value":"d3 = require(\"d3@5\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":83,"value":"import {radio} from \"@jashkenas/inputs\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":64,"value":"import {date} from \"@jashkenas/inputs\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":451,"value":"import {slider} from \"@jashkenas/inputs\"","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W32","stdlib":"1","titling":"auto"}