31:[] 2f:{"title":"Spearman Correlation Coefficient using SpearmanRHO","cells":[{"id":0,"value":"md`# Spearman Correlation Coefficient using SpearmanRHO`","pinned":false,"hidden":false,"mode":"ojs"},{"id":38,"value":"md `I want to compare arrays of a single parameter. I can't find an equivelent of scipy's extensive correlation and signal processing, so I've been experimenting with various smaller libraries. This is one that just does Spearman's Correlation.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":367,"value":"SpearmanRHO = require('https://bundle.run/spearman-rho@1.0.6')","pinned":false,"hidden":false,"mode":"ojs"},{"id":61,"value":"md `I checked the various libraries listed on Tom MacWright's list of JavaScript replacement for Python data tools and couldn't find anything with multiple correlation functions?`","pinned":false,"hidden":false,"mode":"ojs"},{"id":41,"value":"md `------------------------------------`","pinned":false,"hidden":false,"mode":"ojs"},{"id":43,"value":"md `### We'll start with some toy data from the readme. Specifically, two arrays of the same length`","pinned":false,"hidden":false,"mode":"ojs"},{"id":7,"value":"x = [2.0, 3.0, 3.0, 5.0, 5.5, 8.0, 10.0, 10.0]","pinned":false,"hidden":false,"mode":"ojs"},{"id":9,"value":"y = [1.5, 1.5, 4.0, 3.0, 1.0, 5.0, 5.0, 9.5]","pinned":false,"hidden":false,"mode":"ojs"},{"id":50,"value":"md `We'll then create a new SpearmanRHO object and add the two arrays.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":11,"value":"spearmanRHO = new SpearmanRHO(x, y)","pinned":false,"hidden":false,"mode":"ojs"},{"id":52,"value":"md `and finally calculate.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":13,"value":"result = spearmanRHO.calc()","pinned":true,"hidden":false,"mode":"ojs"},{"id":33,"value":"md ` ### Let's check that two identical arrays have a coefficient of 1 by comparing x array and an identical x2 array.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":20,"value":"x2 = [2.0, 3.0, 3.0, 5.0, 5.5, 8.0, 10.0, 10.0]","pinned":false,"hidden":false,"mode":"ojs"},{"id":23,"value":"spearmanRHO_2 = new SpearmanRHO(x, x2)","pinned":true,"hidden":false,"mode":"ojs"},{"id":25,"value":"result2 = spearmanRHO_2.calc()","pinned":false,"hidden":false,"mode":"ojs"},{"id":36,"value":"md `Yes, we get a 1 as expected`","pinned":false,"hidden":false,"mode":"ojs"},{"id":55,"value":"md `------------------------------------------`","pinned":false,"hidden":false,"mode":"ojs"},{"id":57,"value":"md ` ### Let's try this with some real data` ","pinned":false,"hidden":false,"mode":"ojs"},{"id":80,"value":"md `first import some libraries`","pinned":false,"hidden":false,"mode":"ojs"},{"id":370,"value":"md `We'll use d3.js for loading json from github gists`","pinned":false,"hidden":false,"mode":"ojs"},{"id":66,"value":"d3 = require(\"https://d3js.org/d3.v5.min.js\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":372,"value":"md `wellio will be used to work with the json`","pinned":false,"hidden":false,"mode":"ojs"},{"id":79,"value":"wellio = require('wellio@0.1.5/dist/index.js')","pinned":false,"hidden":false,"mode":"ojs"},{"id":84,"value":"md `Then import some well log data. I'm bring in a json of well data, that I've already converted from LAS format to JSON format using wellio.js. Well is 00/01-01-073-05W5/0.las`","pinned":false,"hidden":false,"mode":"ojs"},{"id":64,"value":"log_1 = d3.json(\"https://cdn.rawgit.com/JustinGOSSES/c643e25774aea39556799cc517654d46/raw/db70839e481551dcbadb509fb75c68e3c099de71/00_01-01-073-05W5_0.json\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":86,"value":"md `lets get just the GR log, a single array of floats that come in as strings`","pinned":false,"hidden":false,"mode":"ojs"},{"id":89,"value":"GR1 = wellio.getCurve(log_1,\"GR\") ","pinned":true,"hidden":false,"mode":"ojs"},{"id":92,"value":"md `Now we make a fake one that converts each element of GR1 to a float from a string and adds 2 to the original value.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":95,"value":"function add2(num) {\n return parseFloat(num) + parseFloat(2.0)\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":94,"value":"GR2 = GR1.map(add2)","pinned":false,"hidden":false,"mode":"ojs"},{"id":106,"value":"spearmanRHO_3 = new SpearmanRHO(GR1, GR2)","pinned":false,"hidden":false,"mode":"ojs"},{"id":109,"value":"result3 = spearmanRHO_3.calc()","pinned":false,"hidden":false,"mode":"ojs"},{"id":111,"value":"md `As expected with get a 1 for perfect correlation in the line above, because although the values are different they move in concert exactly as GR2 is just GR1 plus 2.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":117,"value":"md `### Now lets compare two different logs from the same well. These are measuring different properties of the same rocks`","pinned":false,"hidden":false,"mode":"ojs"},{"id":119,"value":"PHIN = wellio.getCurve(log_1,\"PHIN\") ","pinned":false,"hidden":false,"mode":"ojs"},{"id":122,"value":"PHID = wellio.getCurve(log_1,\"PHID\") ","pinned":false,"hidden":false,"mode":"ojs"},{"id":124,"value":"spearmanRHO_4 = new SpearmanRHO(PHIN, PHID)","pinned":false,"hidden":false,"mode":"ojs"},{"id":126,"value":"result_4 = spearmanRHO_4.calc()","pinned":true,"hidden":false,"mode":"ojs"},{"id":130,"value":"md `Let's try comparing GR1 to PHIN for comparison.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":132,"value":"spearmanRHO_5 = new SpearmanRHO(GR1, PHIN)","pinned":false,"hidden":false,"mode":"ojs"},{"id":134,"value":"result_5 = spearmanRHO_5.calc()","pinned":true,"hidden":false,"mode":"ojs"},{"id":137,"value":"md `As expected, there is less correlation between GR1 and PHIN. PHIN and PHID are both Porosity measurements, but GR is gamma ray, which measures the proportion of radioactive minerals, especially clays making it a proxy for % shale.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":140,"value":"md `-------------------------------------------------------------------------------`","pinned":false,"hidden":false,"mode":"ojs"},{"id":143,"value":"md `### Now let's try to find the correlation between the same type of log in different wells. This type of comparison can be used as one part of a program to automate well log correlation.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":148,"value":"md `Importing the new well from similiar location. - 00/02-26-073-05W5/0.las but as JSON`","pinned":false,"hidden":false,"mode":"ojs"},{"id":154,"value":"log_2 = d3.json(\"https://cdn.rawgit.com/JustinGOSSES/9921e53e01c692780ca8a735f1b42cce/raw/3e076174d6198dd2e34b168491605e9645c8179f/00-02-26-073-05W5-0.json\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":164,"value":"GR_B = wellio.getCurve(log_2,\"GR\") ","pinned":false,"hidden":false,"mode":"ojs"},{"id":172,"value":"GR_B.length","pinned":true,"hidden":false,"mode":"ojs"},{"id":175,"value":"GR1.length","pinned":true,"hidden":false,"mode":"ojs"},{"id":180,"value":"GR1_sh = GR1.slice(0,961)","pinned":false,"hidden":false,"mode":"ojs"},{"id":168,"value":"spearmanRHO_6 = new SpearmanRHO(GR1_sh, GR_B)","pinned":false,"hidden":false,"mode":"ojs"},{"id":186,"value":"result_6 = spearmanRHO_6.calc()","pinned":false,"hidden":false,"mode":"ojs"},{"id":188,"value":"md `This is a more negative correlation than looking at PHIN and GR in the log_1 well. Let's visualize it to see to see why.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":190,"value":"md `--------------------------------------------`","pinned":false,"hidden":false,"mode":"ojs"},{"id":194,"value":"md `First lets import vegalite which we will use to display the logs`","pinned":false,"hidden":false,"mode":"ojs"},{"id":192,"value":"vegalite = require(\"@observablehq/vega-lite@0.1\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":203,"value":"md `Next, lets convert the json for each well into an object for visualization that includes the depth array, array of values for a specific curve, and UWI string as a name.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":198,"value":"function convertCurveArToObj(well_log_json,curveString,UWI){\n let depth = wellio.getCurve(well_log_json,\"DEPTH\") \n let curve = wellio.getCurve(well_log_json,curveString) \n let array_of_obj = []\n if (depth.length === curve.length){\n for(let eachPt in curve){\n let obj = {}\n obj[\"depth\"] = depth[eachPt]\n obj[\"curve\"] = curve[eachPt]\n obj[\"curveName\"] = curveString\n obj[\"UWI\"] = UWI\n array_of_obj.push(obj)\n }\n }\n else{\n \tconsole.log(\"depth didn't match curve length\")\n }\n return array_of_obj\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":205,"value":"log_1_GR = convertCurveArToObj(log_1, \"GR\", \"00/01-01-073-05W5/0\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":214,"value":"log_2_GR = convertCurveArToObj(log_2, \"GR\", \"00/02-26-073-05W5/0\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":222,"value":"md `### 00/01-01-073-05W5/0 or log_1`","pinned":false,"hidden":false,"mode":"ojs"},{"id":196,"value":"vegalite({\n \"width\": 820,\n \"height\": 120,\n data: {values: log_1_GR},\n mark: \"line\",\n title:\"GR\",\n color:\"green\",\n encoding: {\n y: {field: \"curve\", type: \"quantitative\", axis: {\n \t// maxExtent:100,\n \t// minExtent:0,\n \t// orient:\"top\"\n \t}\n },\n x: {field: \"depth\", type: \"ordinal\", axis: {\n \tmaxExtent:110,\n \tminExtent:0\n \t// orient:\"left\"\n \t}\n },\n \"color\": {\n \"value\": \"green\"\n }\n }\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":220,"value":"md `### 00/02-26-073-05W5/0 or log_2`","pinned":false,"hidden":false,"mode":"ojs"},{"id":217,"value":"vegalite({\n \"width\": 820,\n \"height\": 120,\n data: {values: log_2_GR},\n mark: \"line\",\n title:\"GR\",\n color:\"green\",\n encoding: {\n y: {field: \"curve\", type: \"quantitative\", axis: {\n \t// maxExtent:100,\n \t// minExtent:0,\n \t// orient:\"top\"\n \t}\n },\n x: {field: \"depth\", type: \"ordinal\", axis: {\n \tmaxExtent:110,\n \tminExtent:0\n \t// orient:\"left\"\n \t}\n },\n \"color\": {\n \"value\": \"green\"\n }\n }\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":225,"value":"md `What we see is there are parts of the log that look similiar, at least roughly, but they are not necessarily at the same depth. Additionally, there are some missing jumps and log shapes that only appear in one well or are different sizes in each well.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":229,"value":"md `### Let's now try to compare a snippet of log_1 and find the piece of log_2 that correlates with it the most.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":277,"value":"function findSliceForDepth(well_log_json,depth){\n let depthCurve = wellio.getCurve(well_log_json,\"DEPTH\")\n function matchDepth(num){\n return parseFloat(num) === parseFloat(depth)\n }\n let _index = depthCurve.findIndex(matchDepth)\n return _index\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":292,"value":"depthCurve = wellio.getCurve(log_1,\"DEPTH\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":288,"value":"log_1_start = findSliceForDepth(log_1,458)","pinned":true,"hidden":false,"mode":"ojs"},{"id":301,"value":"log_1_end = log_1_start + lengthScaler","pinned":false,"hidden":false,"mode":"ojs"},{"id":304,"value":"log_2_start = findSliceForDepth(log_2,416)","pinned":true,"hidden":false,"mode":"ojs"},{"id":308,"value":"log_2_end = log_2_start + lengthScaler","pinned":false,"hidden":false,"mode":"ojs"},{"id":235,"value":"md `Let's modify our convertCurveToObj function to give us only a snipet.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":237,"value":"function convertCurveArToObjSliced(well_log_json,curveString,UWI,start,finish){\n let depth = wellio.getCurve(well_log_json,\"DEPTH\").slice(start,finish) \n let curve = wellio.getCurve(well_log_json,curveString).slice(start,finish)\n let array_of_obj = []\n if (depth.length === curve.length){\n for(let eachPt in curve){\n let obj = {}\n obj[\"depth\"] = depth[eachPt]\n obj[\"curve\"] = curve[eachPt]\n obj[\"curveName\"] = curveString\n obj[\"UWI\"] = UWI\n array_of_obj.push(obj)\n }\n }\n else{\n \tconsole.log(\"depth didn't match curve length\")\n }\n return array_of_obj\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":239,"value":"log_1_GR_slc1 = convertCurveArToObjSliced(log_1, \"GR\",\"00/01-01-073-05W\",log_1_start,log_1_end)","pinned":false,"hidden":false,"mode":"ojs"},{"id":242,"value":"log_2_GR_slc1 = convertCurveArToObjSliced(log_2, \"GR\",\"00/02-26-073-05W5/0\",log_2_start,log_2_end)","pinned":false,"hidden":false,"mode":"ojs"},{"id":244,"value":"vegalite({\n \"width\": 820,\n \"height\": 120,\n data: {values: log_1_GR_slc1},\n mark: \"line\",\n title:\"GR\",\n color:\"green\",\n encoding: {\n y: {field: \"curve\", type: \"quantitative\", axis: {\n \t// maxExtent:100,\n \t// minExtent:0,\n \t// orient:\"top\"\n \t}\n },\n x: {field: \"depth\", type: \"ordinal\", axis: {\n \tmaxExtent:110,\n \tminExtent:0\n \t// orient:\"left\"\n \t}\n },\n \"color\": {\n \"value\": \"green\"\n }\n }\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":248,"value":"vegalite({\n \"width\": 820,\n \"height\": 120,\n data: {values: log_2_GR_slc1},\n mark: \"line\",\n title:\"GR\",\n color:\"green\",\n encoding: {\n y: {field: \"curve\", type: \"quantitative\", axis: {\n \t// maxExtent:100,\n \t// minExtent:0,\n \t// orient:\"top\"\n \t}\n },\n x: {field: \"depth\", type: \"ordinal\", axis: {\n \tmaxExtent:110,\n \tminExtent:0\n \t// orient:\"left\"\n \t}\n },\n \"color\": {\n \"value\": \"green\"\n }\n }\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":253,"value":"log_1_GR_Ar_slc1 = wellio.getCurve(log_1,\"GR\").slice(log_1_start,log_1_end)","pinned":false,"hidden":false,"mode":"ojs"},{"id":256,"value":"log_2_GR_Ar_slc1 = wellio.getCurve(log_2,\"GR\").slice(log_2_start,log_2_end)","pinned":false,"hidden":false,"mode":"ojs"},{"id":259,"value":"spearmanRHO_7 = new SpearmanRHO(log_1_GR_Ar_slc1, log_2_GR_Ar_slc1)","pinned":false,"hidden":false,"mode":"ojs"},{"id":262,"value":"result_7 = spearmanRHO_7.calc()","pinned":false,"hidden":false,"mode":"ojs"},{"id":354,"value":"md `If you move the slider above for \"lengthScaler\" and change the starting point for the comparison via log_1_start and log_2_start, we can see that the starting position and size of the arrays being compared play a big role in the eventual correlation coefficient result. It is easy to vary from -0.33 to 0.49 range`","pinned":false,"hidden":false,"mode":"ojs"},{"id":382,"value":"viewof lengthScaler = html``","pinned":false,"hidden":false,"mode":"ojs"},{"id":385,"value":"lengthScaler","pinned":false,"hidden":false,"mode":"ojs"},{"id":379,"value":"md `---------------------------------------------------`","pinned":false,"hidden":false,"mode":"ojs"},{"id":356,"value":"md `### In a future notebook, I'll work on comparing individual pieces of a well log curve in one well to many pieces in another well. By identifying which snippets have the highest correlation, we'll get to slightly more useful correlation information. There are other techniques for correlation that can be employed as described in this fortran-based program called \"CORRELATOR\".`","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W31","stdlib":"1","titling":"auto"}