30:T79b,md` ### Main workflow 1. Scrape updated USD-TWD exchange from [Investing.com](https://www.investing.com/currencies/usd-twd-historical-data). * Fetch data from previous days from S3 * Input scraped data * Update news values into S3 file 2. Run Rscripts of the statistical models with 2 data inputs, [OWID’s covid update](https://github.com/owid/covid-19-data/blob/master/public/data/owid-covid-data.csv) and the other from step 1. 3. Handling status and notifications including: * Url availability * Update file into S3 * Read, process R model and capture output * Read file and process * Upload to S3 * Report pipeline status ### Data wrangling * One R script aggregates and processes the scraped currency data and [OWID’s covid update](https://github.com/owid/covid-19-data/blob/master/public/data/owid-covid-data.csv) and fills them into our multiple regression formula. * Note that the formula/ model is prebuilt and won’t be updated as data flows in. We have tuned the model against the third wave in the US and the time frame shown in the below chart hence follows. ### Set up The pipeline core runs on python. The statistical models which are made in R run in a subprocess terminal and interact with the pipeline through the standard streams of the system. The code is deployed on a free Heroku dyno properly set to be able to run both python and Rcode. Heroku has default buildpacks for python but none to run R code. Check the code on [github](https://github.com/X80110/MVTEC-Pipeline-dev). ### Specific challenges or difficulties encountered and how they were addressed There are a bunch of possible improvements some of them are specified in the comments in code. Figuring out how the nature of the systems work and how different technologies interact within a system has been challenging but overall a great learning experience offering limitless opportunities to ease work creatively and efficiently. `31:Tb8d,chart = { const aboveUid = DOM.uid("above"); const belowUid = DOM.uid("below"); const svg = d3.create("svg") .attr("viewBox", [0, 0, width, height]) .datum(data); svg.append("g") .call(xAxis); svg.append("g") .call(yAxis); svg.append("clipPath") .attr("id", aboveUid.id) .append("path") .attr("d", d3.area() .curve(curve) .x(d => x(d.date)) .y0(0) .y1(d => y(d.value1))); svg.append("clipPath") .attr("id", belowUid.id) .append("path") .attr("d", d3.area() .curve(curve) .x(d => x(d.date)) .y0(height) .y1(d => y(d.value1))); svg.append("path") .attr("clip-path", aboveUid) .attr("fill", colors[1]) .attr("d", d3.area() .curve(curve) .x(d => x(d.date)) .y0(height) .y1(d => y(d.value0))); svg.append("path") .attr("clip-path", belowUid) .attr("fill", colors[0]) .attr("d", d3.area() .curve(curve) .x(d => x(d.date)) .y0(0) .y1(d => y(d.value0))); // line chart svg.append("path") .attr("fill", "none") .attr("stroke", "#333") .attr("stroke-width", 2) .attr("stroke-linejoin", "round") .attr("stroke-linecap", "round") .attr("d", d3.line() .curve(curve) .x(d => x(d.date)) .y(d => y(d.value0))); svg.append("path") .attr("fill", "none") .attr("stroke", "#0cc3fe") .attr("stroke-width", 2) .attr("stroke-linejoin", "round") .attr("stroke-linecap", "round") // .attr("stroke-dasharray", 4) .attr("d", d3.line() .curve(curve) .x(d => x(d.date)) .y(d => y(d.value1))); // legend svg.call(legend); // tooltip from https://observablehq.com/@d3/line-chart-with-tooltip const tooltip = svg.append("g"); svg.on("touchmove mousemove", function(event) { // svg.on("click", function(event) { // console.log(event); const {date, value0, value1} = bisect(d3.pointer(event, this)[0]); // const eventListener = d3.pointer(event, this); // const xPosition_date = eventListener[0]; // const yPositoin_value = eventListener[1]; // console.log("============"); // console.log(eventListener); // console.log(x.invert(xPosition_date)); // console.log(y.invert(yPositoin_value)); // tooltip // .attr("transform", `translate(${xPosition_date},${yPositoin_value})`) // .call(callout, `${formatValue(y.invert(yPositoin_value))} // ${formatDate(x.invert(xPosition_date))}`); tooltip .attr("transform", `translate(${x(date)},${y(value0)})`) .call(callout, `${formatValue(value0)} ${formatValue1(value1)} ${formatVariance(value0, value1)} ${formatDate(date)}`); }); svg.on("touchend mouseleave", () => tooltip.call(callout, null)); return svg.node(); }32:T625,callout = (g, value) => { console.log("value in callout =============="); console.log(value); if (!value) return g.style("display", "none"); g .style("display", null) .style("pointer-events", "none") .style("font", "10px sans-serif"); const path = g.selectAll("path") .data([null]) .join("path") .attr("fill", "white") .attr("stroke", "black"); const text = g.selectAll("text") .data([null]) .join("text") .call(text => text .selectAll("tspan") .data((value + "").split(/\n/)) .join("tspan") .attr("x", 0) .attr("y", (d, i) => `${i * 1.8}em`) // .style("font-weight", (_, i) => { if (i) { // return null // } else { // return "bold"}}) .style("font-weight", (_, i) => { if (!i || i === 1) { return "bold" } else { return null}}) .style("fill", (_, i) => { if (i === 1) { // color prediction value return "#0bc2ff" } else { return null}}) .text(d => d)); const {x, y, width: w, height: h} = text.node().getBBox(); text.attr("transform", `translate(${-w / 2},${15 - y})`); path.attr("d", `M${-w / 2 - 10},5H-5l5,-5l5,5H${w / 2 + 10}v${h + 20}h-${w + 20}z`); }33:[{"id":"ee840d8edac45668d95b51cc8b285f167cd335b45ccc2cafd5511ad2ddcc548c6c627ceebb06199956f6709dadb6f34b95ab0c0549699d53a44a0a06562152e0","href":"https://static.observableusercontent.com/files/ee840d8edac45668d95b51cc8b285f167cd335b45ccc2cafd5511ad2ddcc548c6c627ceebb06199956f6709dadb6f34b95ab0c0549699d53a44a0a06562152e0","name":"usdtwd_prediction@2.csv","type":"text/csv","lastModified":1610014204621.238,"size":5080},{"id":"d8f7bb5c9d1109d1c8e19b114a324ca61a2f208b5b10efa440b31a24980537b06a422d3cf9c4a196292795276d0920fdc567b5a92030458ca57aa11025e6690a","href":"https://static.observableusercontent.com/files/d8f7bb5c9d1109d1c8e19b114a324ca61a2f208b5b10efa440b31a24980537b06a422d3cf9c4a196292795276d0920fdc567b5a92030458ca57aa11025e6690a","name":"pipeline.002-1.jpeg","type":"image/jpeg","lastModified":1610096267061.504,"size":184059}] 2e:{"title":"PREDICTION MODEL 🔮","cells":[{"id":0,"value":"md`# PREDICTION MODEL 🔮`","pinned":false,"hidden":false,"mode":"ojs"},{"id":867,"value":"md`## 1. Introduction`","pinned":false,"hidden":false,"mode":"ojs"},{"id":968,"value":"md`The downward trend of US dollar to other currencies has lasted for quite some time. What sticks out to us is its exchange rate versus the Taiwan dollar has plummeted to the lowest level in 23 years. Some observers credit that the pandemic was one of the factors that drag the recent rate down. \n\nWhile pandemic rages in every state in the US, Taiwan with a population larger than Florida just ended its longest world covid free record (253 days) with one local case in late December. Hence we are curious about how much one could rely on the covid situation in the US to trade their USD. \n\nFor our statics modeling practice, we approach the question with two figures, daily confirmed cases and deaths of the US. By taking today’s US covid figures, we attempt to predict tomorrow’s USD/TWD rate. \n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1065,"value":"md`## 2a.Data Pipeline`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1050,"value":"image=FileAttachment(\"pipeline.002-1.jpeg\").image()","pinned":false,"hidden":false,"mode":"ojs"},{"id":1082,"value":"$30","pinned":false,"hidden":false,"mode":"ojs"},{"id":1045,"value":"md` ## 2b. Data preparation`","pinned":false,"hidden":false,"mode":"ojs"},{"id":885,"value":"md`\n- *sourceLive* is the csv file hosted on S3 server and updated daily. It is one of the outputs from our heroku app.\n- *sourceArchive* is an attached static csv to serve as a backup file if our data pipeline breaks.\n- function *sourceChecking* determines which source to take by comparing the length of two arrays and status code of our S3 url. Only if it's \\`\\`Response[200]\\`\\` and the requested array is longer than the archive, we will feed *sourceLive* to our visualization.\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":979,"value":"md` #### ↓ Latest data updated on ↓`","pinned":false,"hidden":false,"mode":"ojs"},{"id":950,"value":"d3.max(data, d=> d.date)","pinned":false,"hidden":false,"mode":"ojs"},{"id":1154,"value":"statusCode = fetch('https://mvtec-group3.s3-eu-west-1.amazonaws.com/project/usdtwd_prediction.csv').then(function(response) { \n console.log(response.status); // returns 200 \n return response.status; \n });","pinned":true,"hidden":false,"mode":"ojs"},{"id":816,"value":"sourceArchive = d3.csvParse(await FileAttachment(\"usdtwd_prediction@2.csv\").text())","pinned":false,"hidden":false,"mode":"ojs"},{"id":806,"value":"sourceLive = d3.csv(\"https://mvtec-group3.s3-eu-west-1.amazonaws.com/project/usdtwd_prediction.csv\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":794,"value":"sourceChecking = { \n if (sourceArchive.length < sourceLive.length && statusCode == 200) {\n return sourceLive\n } else {\n return sourceArchive\n }\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":846,"value":"data = sourceChecking.map(d=> {return{\n date: new Date(d.date),\n value0: +d[\"usd2twd_avg\"], // The primary value.\n value1: +d[\"predicted_usd2twd\"] // The secondary comparison value.\n};})","pinned":false,"hidden":false,"mode":"ojs"},{"id":888,"value":"md` ## 3. Visualization and description`","pinned":false,"hidden":false,"mode":"ojs"},{"id":570,"value":"md`### Just how 👍 or 👎 our prediction is? `","pinned":false,"hidden":false,"mode":"ojs"},{"id":532,"value":"viewof parameter = form(html``)","pinned":false,"hidden":false,"mode":"ojs"},{"id":39,"value":"$31","pinned":false,"hidden":false,"mode":"ojs"},{"id":954,"value":"md `#### Features\n- An html input form enables users to fill in a value comparing the gap between the prediction and real rate. \n- One could navigate the gap of each day by hovering over the difference chart. Details in the tooltip include rounded two values, date, and thumbs up/down emoji evaluation on whether the gap falls within users’ desired range. \n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":891,"value":"md`## 4. Conclusions`","pinned":false,"hidden":false,"mode":"ojs"},{"id":1023,"value":"md`Making D3 charts always leaves me with a renewed understanding of how details matter. It’s fast to get the basic lines or shapes corresponding to your data up and running, let alone there are numerous fancy examples you can refer to. \n\nWhen it comes to adjusting fine details like increasing margins between tick text and X-axis or bisecting pointer position value in this chart, I scratched my head and google searched endlessly. Even I found a suitable answer or sample, the works didn’t end there. But with the time and energy I have, I am happy with what I’m able to pull through leaving a few items I’d like to improve soon. \n\nSome thought\n\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":995,"value":"md `\n#### Things to improve/ add\n- The chart was first designed with static data till Dec 21. When the January data is in the axes also are updated and they overlapped with the legend. We will need to think ahead of how the updated data will affect the layout next time. \n- The tooltip is bound with the real rate line only. When the value is larger than the prediction, it blocks the prediction line and the gap. Ideally, the tooltip box should always be either above or below the area but not block it.\n- Include other currencies\n- Add a mini supplementary plot below the line chart to show the US confirmed cases and deaths.\n- Additional residuals VS fitted value plot for easy comparison.\n- Increase margins between tick text and X axis. \n- Extend the width of dashlines.\n\n#### Reference\n- [Difference chart](https://observablehq.com/@d3/difference-chart)\n- [Tooltip](https://observablehq.com/@d3/line-chart-with-tooltip)\n- [Styled Axes](https://observablehq.com/@d3/styled-axes)\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":894,"value":"md`## 5. Implementation details` ","pinned":false,"hidden":false,"mode":"ojs"},{"id":632,"value":"legend = svg => {\n const legendText = [\"Real value\", \"Predicted value\"]\n const legendOrigin = [800,30];\n const labelHeight = 15\n const legend = svg.selectAll(\".legend\")\n .data(legendText)\n .enter();\n \n legend\n .append('text')\n .text(d => d)\n .attr('x', legendOrigin[0] + labelHeight * 1.4)\n .attr('y', (d,i) => legendOrigin[1] + labelHeight + labelHeight * 1.4 *i)\n .style('font-family', 'sans-serif')\n .style('font-size', `${labelHeight-3}px`)\n .attr('fill', d => {if (d == \"Predicted value\"){ return \"#0cc3fe\"} else {return \"black\"}});\n \n legend\n .append('rect')\n .attr('x', legendOrigin[0] - 5)\n .attr('y', (d,i) => legendOrigin[1] + labelHeight * i * 1.4 + 8)\n .attr('width', 20)\n .attr('height', 2.5)\n // .attr('fill', d => {d == \"Predicted value\"? \"blue\" : \"red\"}) #what went wrong? \n .attr('fill', d => {if (d == \"Predicted value\"){ return \"#0cc3fe\"} else {return \"black\"}})\n \n return legend;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":749,"value":"yAxis = svg => svg\n .attr(\"transform\", `translate(${margin.left},0)`)\n .call(d3.axisRight(y)\n // .tickSize(width))\n .tickSize(width - margin.left - margin.right))\n .call(g => g.select(\".domain\")\n .remove())\n .call(g => g.selectAll(\".tick:not(:first-of-type) line\")\n .attr(\"stroke-opacity\", 0.5)\n .attr(\"stroke-dasharray\", \"2,2\"))\n .call(g => g.selectAll(\".tick text\")\n .attr(\"x\", -30)\n .attr(\"dy\", -4))\n .call(g => g.select(\".tick:last-of-type text\").clone()\n .attr(\"x\", 3)\n .attr(\"text-anchor\", \"start\")\n .attr(\"font-weight\", \"bold\")\n .attr(\"font-size\", \"1.3em\")\n .text(data.y)\n .text(\"Conversion rate of 1 USD to TWD\"))","pinned":false,"hidden":false,"mode":"ojs"},{"id":202,"value":"$32","pinned":false,"hidden":false,"mode":"ojs"},{"id":527,"value":"format = d3.timeFormat(\"%Y-%m-%d\");","pinned":false,"hidden":false,"mode":"ojs"},{"id":204,"value":"function formatValue(value) {\n const currency = value.toLocaleString(\"en\", {\n style: \"currency\",\n currency: \"TWD\"\n });\n \n return \"Actual: \" + currency;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":410,"value":"function formatValue1(value) {\n const currency = value.toLocaleString(\"en\", {\n style: \"currency\",\n currency: \"TWD\"\n });\n \n return \"Predicted: \" + currency;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":478,"value":"function formatVariance(val0, val1 ) {\n console.log(\"VAR =========\");\n // console.log(val0);\n // console.log(val1);\n// const currency = value.toLocaleString(\"en\", {\n// style: \"currency\",\n// currency: \"TWD\"\n// });\n const gap = val1 - val0;\n const showGap = +gap.toFixed(3);\n if (Math.abs(gap) > parseFloat(parameter.input)) {\n return (showGap<0?\"\":\"+\") + showGap + \" 👎\";\n } else {\n return (showGap<0?\"\":\"+\") + showGap + \" 👍\";\n }\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":518,"value":"bisect = {\n console.log(\"bisect starts ============\");\n const bisect = d3.bisector(d => d.date).left;\n // const bisect = d3.bisector(d => {d.date}).left;\n // console.log(bisect);\n \n\n return mx => {\n // console.log(\"mx============\");\n // console.log(mx);\n const date = format(x.invert(mx));\n \n// console.log(\"date============\");\n// console.log(bisect2(data, new Date(date)));\n const index = bisect(data, new Date(date));\n \n // console.log(\"index============\");\n // console.log(index);\n \n const a = data[index - 1];\n const b = data[index];\n // console.log(\"a===============\");\n // console.log(a); \n return b && (date - a.date > b.date - date) ? b : a;\n };\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":206,"value":"function formatDate(date) {\n return date.toLocaleString(\"en\", {\n month: \"short\",\n day: \"numeric\",\n year: \"numeric\",\n timeZone: \"UTC\"\n });\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":14,"value":"height = 600","pinned":false,"hidden":false,"mode":"ojs"},{"id":16,"value":"margin = ({top: 20, right: 20, bottom: 30, left: 30})","pinned":false,"hidden":false,"mode":"ojs"},{"id":22,"value":"x = d3.scaleTime()\n .domain(d3.extent(data, d => d.date))\n .range([margin.left, width - margin.right])","pinned":false,"hidden":false,"mode":"ojs"},{"id":24,"value":"y = d3.scaleLinear()\n .domain([\n d3.min(data, d => Math.min(d.value0, d.value1)),\n d3.max(data, d => Math.max(d.value0, d.value1))\n ]).nice(5)\n .range([height - margin.bottom, margin.top])","pinned":false,"hidden":false,"mode":"ojs"},{"id":28,"value":"xAxis = g => g\n .attr(\"transform\", `translate(0,${height - margin.bottom})`)\n .call(d3.axisBottom(x)\n .ticks(width / 80)\n .tickSizeOuter(0)\n .tickSize(0))\n // .call(g => g.select(\".domain\").remove())","pinned":false,"hidden":false,"mode":"ojs"},{"id":151,"value":"colors = [\"rgba(225,225,225, 0.33)\", \"rgba(225,225,225, 0.33)\"]","pinned":false,"hidden":false,"mode":"ojs"},{"id":32,"value":"curve = d3.curveStep\n// curve = d3.curveLinear\n// curve = d3.curveNatural","pinned":false,"hidden":false,"mode":"ojs"},{"id":897,"value":"md`## 6. Appendix`","pinned":false,"hidden":false,"mode":"ojs"},{"id":6,"value":"d3 = require(\"d3@6\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":534,"value":"import {form} from \"@mbostock/form-input\"","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W33","stdlib":"1","titling":"auto"}