31:T405,md`# Social activity visualization Let's explore a user's activity on an internet forum. We have the API to fetch the user's activity like posts and comments. For every activity, we can see the date and time when it was created. We can also see the rating – a number that shows how other users rated a given post or comment. Now, let's draw a chart that shows the date on the *x*-axis and the time on the *y*-axis. Let's draw the user's activities on the chart. Every point will be a post or comment. A big green point means an activity with a positive rating, a smaller gray point means an activity with zero-rating and a big red point is an activity with negative rating:`32:T45b,{ const height = 600 const margin = { top: 20, right: 30, bottom: 30, left: 40 } const svg = d3.select(DOM.svg(width, height)) const x = d3.scaleTime() .domain(d3.extent(data, d => d.date)) .range([margin.left, width - margin.right]) const y = d3.scaleTime() .domain(d3.extent(data, d => d.time).reverse()) .range([height - margin.bottom, margin.top]) const xAxis = g => g .attr('transform', `translate(0,${height - margin.bottom})`) .call(d3.axisBottom(x)) const yAxis = g => g .attr('transform', `translate(${margin.left},0)`) .call(d3.axisLeft(y).ticks(24, '%H:00')) svg.append('g').call(xAxis) svg.append('g').call(yAxis) const positive = '#00C853' const neutral = '#757575' const negative = '#FF5722' svg.append('g') .selectAll('circle') .data(data) .enter() .append('circle') .attr('cx', d => x(d.date)) .attr('cy', d => y(d.time)) .attr('fill', d => d.rating > 0 ? positive : (d.rating < 0 ? negative : neutral)) .attr('r', d => d.rating == 0 ? 1.5 : 2); return svg.node() }33:T6a8,md`From this chart we can see some interesting patterns: * The user wrote his first comments in 2014 but was not very active. * For some reason, he started to post something at the end of 2014 and beginning of 2015. * From the beginning of 2016, the user started to post and comment more. * The red lines show us where the user had some *hot* discussions with others. * We can see when the user is usually going to sleep and on which day he stayed late. The thing about the sleep patterns and the user's presence on the forum looks interesting. Let's draw a chart, with the same axes, but draw a line if it is possible to detect the user's absence on the forum during the day. Of course, we can do that only when the user posts something for several days in a row: say, the user posted on Monday, Tuesday, Wednesday, and Friday. We can *guess* when the user slept on Tuesday because there are activities on Monday and Wednesday. But if there is a comment, say, at 23:00 on Wednesday and then 7:00 on Friday, we can't see if the user went to bed on Wednesday 23:30 or Thursday 11:45. Anyway, let's draw a chart. The grey lines are for inactivities that are longer than 8 hours, orange – more than 6 hours and less than 8, red – less than 6. As we can see, there are only two times when the user didn't sleep long enough:` 34:T5f2,{ const height = 600 const margin = { top: 20, right: 30, bottom: 30, left: 40 } const svg = d3.select(DOM.svg(width, height)) const x = d3.scaleTime() .domain(d3.extent(data, d => d.date)) .range([margin.left, width - margin.right]) const y = d3.scaleTime() .domain(d3.extent(data, d => d.time).reverse()) .range([height - margin.bottom, margin.top]) const xAxis = g => g .attr('transform', `translate(0,${height - margin.bottom})`) .call(d3.axisBottom(x)) const yAxis = g => g .attr('transform', `translate(${margin.left},0)`) .call(d3.axisLeft(y).ticks(24, '%H:00')) svg.append('g').call(xAxis) svg.append('g').call(yAxis) const positive = '#B0BEC5' const neutral = '#FF9800' const negative = '#FF5722' svg.append('g') .selectAll('line') .data(sleeps.sort((a, b) => a.hours > b.hours ? -1 : 1 )) .enter() .append('line') // .attr('x1', d => x(d.end)) .attr('y1', d => { const start = new Date(d.start) start.setFullYear(1900, 0, 1) return y(new Date(start)) }) .attr('x2', d => x(d.end)) .attr('y2', d => { const end = new Date(d.end) end.setFullYear(1900, 0, 1) return y(new Date(end)) }) .attr('stroke', d => d.hours > 8 ? positive : (d.hours < 6 ? negative : neutral)) .attr('stroke-width', d => d.hours > 8 ? 0.7 : (d.hours < 6 ? 2 : 1.5)) return svg.node() }35:T5fd,{ const height = 600 const margin = { top: 20, right: 30, bottom: 30, left: 40 } const svg = d3.select(DOM.svg(width, height)) const x = d3.scaleTime() .domain(d3.extent(data, d => d.date)) .range([margin.left, width - margin.right]) const max_rating = d3.max(ratings, r => Math.max(r.positive, r.negative)) const y = d3.scaleLinear() .domain([-d3.max(ratings, r => r.negative), d3.max(ratings, r => r.positive)]).nice() .range([height - margin.bottom, margin.top]) const xAxis = g => g .attr('transform', `translate(0,${height - margin.bottom})`) .call(d3.axisBottom(x)) const yAxis = g => g .attr('transform', `translate(${margin.left},0)`) .call(d3.axisLeft(y).ticks(10)) svg.append('g').call(xAxis) svg.append('g').call(yAxis) const positive = '#00C853' const negative = '#FF5722' svg.append('g') .selectAll('line') .data(ratings) .enter() .append('line') // .attr('x1', r => x(r.date)) .attr('y1', r => y(0)) .attr('x2', r => x(r.date)) .attr('y2', r => y(r.positive)) .attr('stroke', positive) svg.append('g') .selectAll('line') .data(ratings) .enter() .append('line') // .attr('x1', r => x(r.date)) .attr('y1', r => y(0)) .attr('x2', r => x(r.date)) .attr('y2', r => y(r.negative * -1)) .attr('stroke', negative) return svg.node() }36:T44d,md`## Comparing two users Some people create two or more accounts. We will not discuss the reasons for this but will try to visually compare to user activities and see if they belong to a single person. We draw the activities of two users on a single chart. Each activity is represented by a short vertical line, red for the first user and blue for the second. If a person posts from two accounts almost in the same time, this may look like . The lines will overlay each other and we will easily notice the common patterns in social activities of these two users: `37:T6ed,{ const height = 600 const margin = { top: 20, right: 30, bottom: 30, left: 40 } const svg = d3.select(DOM.svg(width, height)) const x = d3.scaleTime() .domain(d3.extent(diff0, d => d.date)) .range([margin.left, width - margin.right]) const y = d3.scaleTime() .domain(d3.extent(diff0, d => d.time).reverse()) .range([height - margin.bottom, margin.top]) const xAxis = g => g .attr('transform', `translate(0,${height - margin.bottom})`) .call(d3.axisBottom(x)) const yAxis = g => g .attr('transform', `translate(${margin.left},0)`) .call(d3.axisLeft(y).ticks(24, '%H:00')) svg.append('g').call(xAxis) svg.append('g').call(yAxis) const shift = 3 const horizontal_shift = 1 const color_a = '#F44336' const color_b = '#2196F3' const opacity = .9 const stroke_width = 2 svg.append('g') .selectAll('line') // .data(diff0.filter(d => d.user === 'a')) .enter() .append('line') .attr('x1', d => x(d.date) + horizontal_shift) .attr('y1', d => y(d.time) - shift) .attr('x2', d => x(d.date) + horizontal_shift) .attr('y2', d => y(d.time) + shift) .attr('stroke', color_a) .attr('opacity', opacity) .attr('stroke-width', stroke_width) svg.append('g') .selectAll('line') // .data(diff0.filter(d => d.user === 'b')) .enter() .append('line') .attr('x1', d => x(d.date)) .attr('y1', d => y(d.time) - shift) .attr('x2', d => x(d.date)) .attr('y2', d => y(d.time) + shift) .attr('stroke', color_b) .attr('opacity', opacity) .attr('stroke-width', stroke_width) return svg.node() }38:T6ed,{ const height = 600 const margin = { top: 20, right: 30, bottom: 30, left: 40 } const svg = d3.select(DOM.svg(width, height)) const x = d3.scaleTime() .domain(d3.extent(diff1, d => d.date)) .range([margin.left, width - margin.right]) const y = d3.scaleTime() .domain(d3.extent(diff1, d => d.time).reverse()) .range([height - margin.bottom, margin.top]) const xAxis = g => g .attr('transform', `translate(0,${height - margin.bottom})`) .call(d3.axisBottom(x)) const yAxis = g => g .attr('transform', `translate(${margin.left},0)`) .call(d3.axisLeft(y).ticks(24, '%H:00')) svg.append('g').call(xAxis) svg.append('g').call(yAxis) const shift = 3 const horizontal_shift = 1 const color_a = '#F44336' const color_b = '#2196F3' const opacity = .9 const stroke_width = 2 svg.append('g') .selectAll('line') // .data(diff1.filter(d => d.user === 'a')) .enter() .append('line') .attr('x1', d => x(d.date) + horizontal_shift) .attr('y1', d => y(d.time) - shift) .attr('x2', d => x(d.date) + horizontal_shift) .attr('y2', d => y(d.time) + shift) .attr('stroke', color_a) .attr('opacity', opacity) .attr('stroke-width', stroke_width) svg.append('g') .selectAll('line') // .data(diff1.filter(d => d.user === 'b')) .enter() .append('line') .attr('x1', d => x(d.date)) .attr('y1', d => y(d.time) - shift) .attr('x2', d => x(d.date)) .attr('y2', d => y(d.time) + shift) .attr('stroke', color_b) .attr('opacity', opacity) .attr('stroke-width', stroke_width) return svg.node() }39:T7ad,sleeps = { const _24h = 1000 * 60 * 60 * 24 const result = [] const by_date = {} const push = (key, record) => { if(by_date[key]) { by_date[key].push(record) } else { by_date[key] = [record] } } data.forEach((d, i) => { if (i === 0) return const key_start = new Date(data[i - 1].date).toISOString() const key_end = new Date(d.date).toISOString() const start = new Date(data[i - 1].created) const end = new Date(d.created) const distance = i < data.length - 1 ? new Date(d.created) - new Date(data[i - 1].created) : _24h * 2 const record = { start: start, end: end, distance: distance, hours: distance / 1000 / 60 / 60 } push(key_start, record) push(key_end, record) }) const keys = Object.keys(by_date) const find_max_record = (records) => { let record = null for(let i = 0; i < records.length; i++) { if(!record || record.distance < records[i].distance) { record = records[i] } } return record } const max_spans = [] for(let i = 0; i < keys.length; i++) { let record = find_max_record(by_date[keys[i]]) if(record.distance < _24h) { max_spans.push(record) } } max_spans.forEach((d, i) => { if(i === 0 || d.start - max_spans[i - 1].start !== 0) { if(d.start.toDateString() !== d.end.toDateString()) { const midnight = new Date(new Date(d.end).setHours(0, 0, 0, 0)) result.push({ start: d.start, end: new Date(new Date(d.start).setHours(23, 59, 59, 0)), distance: d.distance, hours: d.hours }) result.push({ start: new Date(new Date(d.end).setHours(0, 0, 0, 0)), end: d.end, distance: d.distance, hours: d.hours }) } else { result.push(d) } } }) return result//.filter(d => d.hours < 6) }3a:[] 2f:{"title":"Social activity visualization","cells":[{"id":0,"value":"$31","pinned":false,"hidden":false,"mode":"ojs"},{"id":645,"value":"$32","pinned":false,"hidden":false,"mode":"ojs"},{"id":927,"value":"$33","pinned":false,"hidden":false,"mode":"ojs"},{"id":648,"value":"$34","pinned":false,"hidden":false,"mode":"ojs"},{"id":964,"value":"md`Now let's see how many positive and negative ratings the user got on each day:`","pinned":false,"hidden":false,"mode":"ojs"},{"id":753,"value":"$35","pinned":false,"hidden":false,"mode":"ojs"},{"id":966,"value":"$36","pinned":false,"hidden":false,"mode":"ojs"},{"id":917,"value":"$37","pinned":false,"hidden":false,"mode":"ojs"},{"id":1016,"value":"md`The common patterns are clear here. Here is another comparison, a person has created another user account but keeps posting from the first one from time to time:`","pinned":false,"hidden":false,"mode":"ojs"},{"id":860,"value":"$38","pinned":false,"hidden":false,"mode":"ojs"},{"id":69,"value":"d3 = require(\"https://d3js.org/d3.v5.min.js\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":75,"value":"data = {\n const activities = await d3.json('https://gist.githubusercontent.com/romaklimenko/97d97f5d1b03d43049bdb458b1e98db7/raw/911eae13e7d3e46e04b9a119b73b7b905845d27c/activities.json')\n return activities.map(a => {\n const created = new Date(a.created)\n const date = new Date(created)\n date.setHours(0, 0, 0)\n a.date = date\n const time = new Date(1900,0)\n time.setHours(created.getHours(), created.getMinutes())\n a.time = time\n return a\n })\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":856,"value":"diff0 = {\n const url_a = 'https://gist.githubusercontent.com/romaklimenko/81af5f3133d5c3d6844f6b1230bf9dd6/raw/a5de4530a1200c0d2ebd171226582c6c0d4a01b3/Iamalive-activities.json'\n const url_b = 'https://gist.githubusercontent.com/romaklimenko/7d69e40578bc6f8d5d4b491135b70464/raw/c180782f8440f4e83ab82e561336a86c074ff8e9/Dean_Winchester-activities.json'\n\n const activities_a = await d3.json(url_a)\n const activities_b = await d3.json(url_b)\n \n const result = activities_a.map(a => Object.assign({ user: 'a' }, a))\n .concat(activities_b.map(b => Object.assign({ user: 'b' }, b)))\n .map(d => {\n const created = new Date(d.created)\n const date = new Date(created)\n date.setHours(0, 0, 0)\n d.date = date\n const time = new Date(1900,0)\n time.setHours(created.getHours(), created.getMinutes())\n d.time = time\n return d\n })\n \n result.sort((a, b) => new Date(a.created) - new Date(b.created) > 0 ? 1 : -1)\n \n return result\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":910,"value":"diff1 = {\n const url_a = 'https://gist.githubusercontent.com/romaklimenko/4375775cb68e564f3c758eaeb52dfb92/raw/0426dfe3c4c767c0debb4594fb8b79e2d891de7d/bitl-activities.json'\n const url_b = 'https://gist.githubusercontent.com/romaklimenko/ace9cc38804d3b70f2760b3f3eca3c58/raw/e099f2013040cd4e76d6251bdbf25cd75171f74b/unlaba-activities.json'\n \n \n const activities_a = await d3.json(url_a)\n const activities_b = await d3.json(url_b)\n \n const result = activities_a.map(a => Object.assign({ user: 'a' }, a))\n .concat(activities_b.map(b => Object.assign({ user: 'b' }, b)))\n .map(d => {\n const created = new Date(d.created)\n const date = new Date(created)\n date.setHours(0, 0, 0)\n d.date = date\n const time = new Date(1900,0)\n time.setHours(created.getHours(), created.getMinutes())\n d.time = time\n return d\n })\n \n result.sort((a, b) => new Date(a.created) - new Date(b.created) > 0 ? 1 : -1)\n \n return result\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":642,"value":"$39","pinned":false,"hidden":false,"mode":"ojs"},{"id":749,"value":"ratings = {\n const _24h = 1000 * 60 * 60 * 24\n const result = []\n \n let date = new Date(data[0].date)\n let positive = 0\n let negative = 0\n \n data.forEach((d, i) => {\n if(new Date(d.date) - new Date(d) !== 0) {\n result.push({\n date: date,\n positive: positive,\n negative: negative\n })\n date = new Date(d.date)\n positive = 0\n negative = 0\n if(d.rating > 0) {\n positive += d.rating\n }\n else if(d.rating < 0){\n negative += -d.rating\n }\n }\n })\n \n return result//.filter(d => d.hours < 6)\n}","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W3a","stdlib":"1","titling":"auto"}