2c:T40a,md`## OK, if Circular Diagrams, ... then, in order to emulate this approach, the first step is to enumerate the entities and the relationships. Based on that catalog we can think about how to arrange the entities and call out the relationships. #### entities - repositories - views - predicates - variables - predicates in rdf statements - federation - subqueries - external pull resources - external push resources - reports (html, diagrams, data) - inference capabilities - model creation sources - reified models #### relationships - import - contained documents : typed-stream -> query -> dataset - data streams : typed-stream -> query -> dataset - export request - contained documents : dataset -> typed document - data streams : dataset -> query -> typed stream - query request - document results : dataset -> query -> typed document - result streams : dataset -> query -> typed stream - sub-query request : dataset -> query -> solutions (?) - external request : source -> query -> solutions (?) `2d:T9a7,md` ## ivan notes just putting some small thoughts down that i still need to process - ### outermost ring - encodings ### next ring - resources and agents (external resources) - should be distinguished by some sort of level of trust - internally curated repositories (should be visually disnguished native things) - digital twins of sensors - curated data sets /derived data sets - internally externalized api - external API (authincated/private) - external repository (e.g. dppedia) - other open api ### next ring - views with lines to resources (visible diplay of compositions sources) - federation is displayed as a connection type between views (visble lines of a diffrent kind) ### next ring - rules (lines to views and to resources) ### next ring (perhaps rules and models should share a ring as do agents and resources) - models ### interior ring - environments (deployment defnitions) - lines should be most complex outward as they would be the physical points of integration for the logiical components - inner linkages represnet environment IO relationships any thing that sits on any of the rings could have a height which indicates temporal versions or this dimnesions could be flatnned by using stacked histograms as visuulazations of versions or windows? - dependency relationships over time - annotations, comments Scale streching is an interesting thing to explore as a zoom capability would having rings like this https://observablehq.com/@d3/zoomable-sunburst be too mind bendy as a way to explore a ring. --- Can this be reduced to: - outer ring is external resources - virtual repositories - sensors? - ml processes? - web page interfaces - intermediate ring(s) views w/respective predicates arranged as "genomic positions" - center is internal resources - repositories - ml processes? - sensors? Is it possible to make it into a toy by adding ways to interactively manipulate those elements - not just highlighting, but adding, removing and changing their properties/content. This should include - indirect manipulation, for things like adding/removing predicates from views, which then changes the display. - direct manipulation - composing views to join them - splitting views into components - attaching views to sources/destinations - ? The indirect manipulation would be through secondary pop-up dialogs and/or windows while the direct manipulation would be in-place. `2e:T9be,md` This includes just the four sorts of things which determine the course of processes the service - predicates, classes, repositories and views. There may be specific variations to individual of those, but that taxonomy suffices to capture their general nature. If they are compared to the entites in a [circos](https://www.circos.ca) diagram or in the cardcaptorsakura presentation, adaquete distinctions emerge to demonstrate, that neither is appropriate as a direct basis to present and manipulate service instances - circos concerns ways to depict the linear relations among cytogenetic bands, genes, chromosomes and genomes in a compact manner by arranging them circularly, locating specific information radially and connecting locations to call out relationships. - cardcaptorsakura depicts the temporal relations among a story's episodes in a circle in order, locates the specific plot properies radially and places the characters centrally in order to depict their role(s) by connecting to episodes and plot features. An interface to the service requires some other arrangement, as the relationships are those of a acyclic graph, rather than a line. - predicate are part of multiple classes, views and repositories. - classes are concretely present in repository, but abstractly present in multiple - views are associated with one repository, but can reference multiple other in addition - repositories comprise individual concrete statements and terms - among them predicates, as well as classes, but share them abstractly and are referenced by views, but otherwise stand along. this suggests an arrangement where the ideograms - from circos terminology, depict views. within each onee, the respective preciates are arranges linearly, grouped according to class - analogous to circos' cytogenetic bands. the respecive rdial information could be the respective dataset cardinality or the cardinality specific to the view. An individual circle for each repository would be placed in the inner region if the repository is owned by the account or attached to an outer ring if it is an external resource. links would indicate relations between the views and repositories and between a view and any sub-views. The first step is to generate this model within the service from the entites already present. Once that exists, evolve the interaction model (with remote http operation resources or just local event handelers) and judge wheter to generate the svg with d3 or server-side. `2f:Tc96,{ var frame = document.createElement("div"); var canvas = document.createElement("canvas"); var controls = document.createElement("div"); var account = document.createElement("text"); var print = document.createElement("button"); var refresh = document.createElement("button"); var displayModel = function(model) { let ctx = new OldModel.ClickableRenderingContext(canvas); let width = canvas.width; let height = canvas.height; canvas.addEventListener("click", function(event) { ctx.handleClick(event); }); console.log("ctx: ", ctx); OldModel.Model.bind(model); console.log("new model: ", model); model.position = new OldModel.Model.Position(width / 5, height / 4); var radius = model.radius; let scale = Math.min(width, height) / (5 * radius); ctx.save(); ctx.clear(); ctx.scale(scale, scale); ctx.lineWidth = .5; ctx.imageSmoothingEnabled = false; model.render(ctx); ctx.restore(); }; var printModel = function() { console.log("clicked"); let url = canvas.toDataURL(); var newWindow = window.open("", `${hostName}/${account.innerText}`); var d = newWindow.document; d.close(); var title = document.createElement("title"); title.innerText = `${hostName}/${accountName}`; d.head.appendChild(title); d.body.appendChild(document.createElement('img')).src = url; }; var refreshModel = function() { var location = `https://${hostName}/system/accounts/${account.innerText}`; ResourceAccess.getResource( location, { Accept: "application/json" }, displayModel ); }; print.style.marginTop = "4px"; print.style.marginRight = "3px"; print.style.width = "26px"; print.style.height = "26px"; print.style.border = "solid 1px #a0a0a0"; print.style.backgroundImage = "url('https://dydra.com/icons/printer.svg')"; print.style.display = "inline-block"; refresh.style.marginTop = "4px"; refresh.style.marginRight = "3px"; refresh.style.width = "26px"; refresh.style.height = "26px"; refresh.style.border = "solid 1px #a0a0a0"; refresh.style.backgroundImage = "url('https://dydra.com/icons/rotate.svg')"; refresh.style.display = "inline-block"; account.innerText = accountName; account.contentEditable = true; account.style.display = "inline-block"; account.style.position = "relative"; account.style.top = "-6px"; account.style.width = "150px"; account.style.textAlign = "right"; account.style.border = "solid 1px #a0a0a0"; account.style.paddingLeft = "4px"; account.style.paddingRight = "4px"; frame.appendChild(canvas); controls.appendChild(print); controls.appendChild(refresh); controls.appendChild(account); frame.appendChild(controls); canvas.width = window.innerWidth; canvas.height = 600; frame.style.borderLeft = "solid 1px #a0a0a0"; frame.style.borderRight = "solid 1px #a0a0a0"; frame.style.padding = "2px"; print.onclick = function() { printModel(); }; refresh.onclick = function() { refreshModel(); }; //refreshModel(); //console.log("refresh ...."); // archiving in place frame.innerText = "[... archived older version of the account presentation]"; return frame; }30:T5a8,{ var frame = document.createElement("div"); var presentation = document.createElement("div"); var displayModel = function(model) { let oldSvg = presentation.querySelector('svg'); if (oldSvg) { presentation.removeChild(oldSvg); } let context = new Presentation.PresentationContext(presentation); let svg = context.root; OldModel.Model.bind(model); console.log("new model: ", model); model.position = new OldModel.Model.Position(0, 0); context.present(model); var bbox = svg.getBBox(); context.viewbox = `${bbox.x}, ${bbox.y}, ${bbox.width}, ${bbox.height}`; context.lineWidth = .5; context.fontFamily = "serif"; context.fontSize = "9pt"; window.model = model; window.context = context; }; var refreshModel = function(location) { console.log("refreshing", location); ResourceAccess.getResource( location, { Accept: "application/json" }, displayModel ); }; presentation.width = window.innerWidth; presentation.height = 600; frame.style.borderLeft = "solid 1px #a0a0a0"; frame.style.borderRight = "solid 1px #a0a0a0"; frame.style.padding = "2px"; frame.appendChild(presentation); if (accountName) { // refreshModel(`https://${hostName}/system/accounts/${accountName}`); // console.log("refresh ...."); } frame.innerText = "[... another archived older version of the account presentation]"; return frame; }31:T530,simpleRepositoryDisplay = { let div = document.createElement("div"); let label = document.createElement("div"); let viewEditor = simpleViewListEditor.html; let predicateEditor = simplePredicateListEditor.html; let diagram = simpleRepositoryDiagram.content; let iframe = document.createElement("iframe"); viewEditor.style.backgroundColor = "rgb(255,255,255,.5)"; viewEditor.style.border = "1px dotted gray"; viewEditor.style.overflow = "hidden"; viewEditor.style.resize = "both"; viewEditor.style.width = "auto"; viewEditor.style.position = "absolute"; viewEditor.style.top = "24px"; viewEditor.style.left = "4px"; predicateEditor.style.backgroundColor = "rgb(255,255,255,.5)"; predicateEditor.style.border = "1px dotted gray"; predicateEditor.style.overflow = "hidden"; predicateEditor.style.resize = "both"; predicateEditor.style.width = "auto"; predicateEditor.style.position = "absolute"; predicateEditor.style.top = "324px"; predicateEditor.style.left = "4px"; iframe.style.width = "100%"; iframe.style.height = "600px"; label.innerText = `view/pattern diagram for ${hostName}:${accountName}/${repositoryName}`; div.appendChild(viewEditor); div.appendChild(predicateEditor); div.appendChild(label); div.appendChild(iframe); iframe.srcdoc = diagram; return div; }32:T4eb,repositoryPresentation = deploy("service-interface", async (req, res) => { let headers = req.headers; let query = req.query; let hostName = query['hostName']; let accountName = query['accountName']; let repositoryName = query['repositoryName']; let token = query['token']; let frame = document.createElement("div"); frame.width = 1280; frame.height = 1024; if ( hostName && accountName && repositoryName && repositoryPresentationFrame ) { function respondWithContent() { let svg = frame.querySelector('svg'); let content = new XMLSerializer().serializeToString(svg); res.header("content-type", "image/svg+xml"); res.send(content); } var repositoryCell = new SERVICE.RepositoryCell(null, { view: frame, D3: D3, COLA: COLA, hostName: hostName, repositoryId: `${accountName}/${repositoryName}`, authentication: token, continuation: respondWithContent }); } else { let content = `cannot interpret: ${req.url}`; res.header("content-type", "text/plain"); res.send(content); } // https://endpointservice.web.app/notebooks/@datagenous/sow-20210305-visible-workflows/deployments/service-interface/cells/accountPresentation?... })33:T4d9, This diagram depicts the `application/json` response to requests for `https://${hostName}/system/accounts/${accountName}`. That response represents a graph model of the components in an account - the account, encompassed by the outer circular region - its repositories, the red elements - their concept patterns, the orange discs - their analysis and output, import, and transformation views - the colored segment on the outer ring - their projection patterns, the golden discs - their import pattern, the green disks This model correlates an inside-out, "formal concept analysis" of the star graphs of each repository, with an outside-in catalog of all defined views to yield the graph of relations between the hierarchy of basic patterns implicit in the repositories' data and the the analogous patterns which appear in the views which provide the means to ingest and present that data. Each repository's views are examined to distinguish their intended roles in get, post, and update operations. The client presentation employs the model to depict connections between data and view patterns and among repositories and [presents](https://observablehq.com/@datagenous/resource-presentation-examples) as a circular form. 34:T87c,spreadsheetExample = md`#### projective spreadsheets : Introduce a collection of spreadsheets which the service -- projectively aligns via import and prepocessing -- connects with reporting or visualization steps based on a description of dimensions and media type. 1. Accept a drop of directory or a collection of files: * most of the files contain "input" data, * one file can be an example of output - csv, xsl, svg, or dot. - dot : parse the dot file ([graphlib-dot](https://github.com/dagrejs/graphlib-dot), [dotparser](https://github.com/anvaka/dotparser), [dotgraph](https://github.com/siefkenj/dotgraph), ...) extract the graph root node and link label values - json-based graphics employ for the graph model ([sigmajs](http://sigmajs.org/)) some sort of schema which specifies the attribute to identify labels. frequent are the "node" and "link" attributes for the respective objects and the "label" attribute for labels. * one file can be a schema declaration - an annotated [spreadsheet/csv](https://docs.google.com/spreadsheets/d/1y6_grkqaY-bXbfbEjYMavedrrxg23THrqHu7qQ96OJs/edit?ts=60796aa4#gid=1017225213) - a json-ld context with one or more @id stipulations 1. Construct appropriate BGPs which reify the node and link relationships extracted from the presentation documents. 1. Identify candidate flows * associate each input file with one or more import views. - absent known views, generate as necessary, using the account location as the base iri, * collate the views according to target repository, * Use the prospective BGPs to associate the prototype result documents with projection views * Search for backwards from the projection views to their repositories, through the BGPs to locate direct projection views for the input BGPs or join views which cover thepath from input to output. - recognize correlations as per the schema and/or the result - integrate intermediate update and construct views where necessary - display completely bound flows _and_ incomplete flows * if just one flow is instantiated, execute it and display the results. `35:Taa9,md`### gestures The diagram captures the central components of the initial workflow, above. Given this visual representation, numerous opportunities exist to manage it interactively: The implemented mouse gestures afford means to examine and combine them. | manipulate | target | action (_NYI_) | effect | |--------------------|-----------|--------|-----------------------| | repository pattern | view ring | d&d | create a new get view | | repository pattern | view | d&d | augment the view | | file | post view | d&d |import the file via the view | | file | "^" icon | d&d | extract correlate that import view headers and indicate matches. | | graph pattern | | double-click | present the constituent predicates| Numerous other gestures suggest themselve. In this list, the stricken items have been reslized, while the remainder reveal an alert to describe the interaction which still lacks definition. - drop a pattern on the view ring to create or augment a view - drag a pattern or an entire view from another realm and drop in on the service ring. - open a repository - as a window to manage it - as a presentation of the view results - as another realm in this presentation where the inner element are its patterns - hover over a pattern to highlight - other patterns with shared predicates - suggest paths by showing links to other patterns - other patterns for which the subject can appear as an object in this pattern to compose a path - drag a pattern over other patterns in the same repository to highlight the views which combine them. - drag a pattern over other patterns with which it is not yet combined highlights those with which it composes a path - drop a pattern over another pattern either highlights a view which combined them or creates a new view which joins them - explode an expression to expose its composed patterns - mute a pattern (filter?) - on hover display diplay modes (run vs configure now governed by a switch) - view onclick run : open an html page with results - view onclick+alt : open a page to edit the view - repository onclick+alt : open a page to edit the repository (via modal 'open') ... - onclick configure - link to query (eventually to detail circle diagram of class) - onclick run for external class -link to results page - onclick configure run for external class - link to paramterization UX `36:Tf9c,{ var text = ` digraph "E5A46CD4-A53D-11EB-9A29-0CC47A92447E" { rankdir=LR; fontname=courier;node [fontname=courier];edge [fontname=courier];"7648323" [label="/jhacker"];"7648324" [label="j.q. hacker"];"7648323" -> "7648324" [label=name];"7648326" [label="/Person"];"7648323" -> "7648326" [label=class];"7648325" [label="hacker@example.org"]; "7648323" -> "7648325" [label=mbox]; "7648323" -> "7648325" [label=other]; "7648323" -> "7648324" [label=name];"7648323" -> "7648326" [label=class];"7648323" -> "7648325" [label=mbox];"7648323" -> "7648324" [label=name];"7648323" -> "7648326" [label=class];"7648323" -> "7648325" [label=mbox];"7648323" -> "7648324" [label=name];"7648323" -> "7648326" [label=class];"7648323" -> "7648325" [label=mbox];"7648323" -> "7648324" [label=name];"7648323" -> "7648326" [label=class];"7648323" -> "7648325" [label=mbox];"7648323" -> "7648324" [label=name];"7648323" -> "7648326" [label=class];"7648323" -> "7648325" [label=mbox];"7648323" -> "7648324" [label=name];"7648323" -> "7648326" [label=class];"7648323" -> "7648325" [label=mbox];"7648323" -> "7648324" [label=name];"7648323" -> "7648326" [label=class];"7648323" -> "7648325" [label=mbox];"7648323" -> "7648324" [label=name];"7648323" -> "7648326" [label=class];"7648323" -> "7648325" [label=mbox];"7648323" -> "7648324" [label=name];"7648323" -> "7648326" [label=class];"7648323" -> "7648325" [label=mbox];"7648323" -> "7648324" [label=name];"7648323" -> "7648326" [label=class];"7648323" -> "7648325" [label=mbox];"7648323" -> "7648324" [label=name];"7648323" -> "7648326" [label=class];"7648323" -> "7648325" [label=mbox];"7648323" -> "7648324" [label=name];"7648323" -> "7648326" [label=class];"7648323" -> "7648325" [label=mbox];"7648323" -> "7648324" [label=name];"7648323" -> "7648326" [label=class];"7648323" -> "7648325" [label=mbox];"7648323" -> "7648324" [label=name];"7648323" -> "7648326" [label=class];"7648323" -> "7648325" [label=mbox];"7648323" -> "7648324" [label=name];"7648323" -> "7648326" [label=class];"7648323" -> "7648325" [label=mbox];"7648323" -> "7648324" [label=name];"7648323" -> "7648326" [label=class];"7648323" -> "7648325" [label=mbox];"7648323" -> "7648324" [label=name];"7648323" -> "7648326" [label=class];"7648323" -> "7648325" [label=mbox];"7648323" -> "7648324" [label=name];"7648323" -> "7648326" [label=class];"7648323" -> "7648325" [label=mbox];"7648323" -> "7648324" [label=name];"7648323" -> "7648326" [label=class];"7648323" -> "7648325" [label=mbox];"7648302" [label="/basePerson"];"7648303" [label="Base Person"];"7648302" -> "7648303" [label=name];"843" [label="foaf:Person"];"7648302" -> "843" [label=class];"7648304" [label=label1];"7648302" -> "7648304" [label=mbox];}`; var headers = GD.GraphvizData.computeHeaders(text, { parser: ChartGraphsDotParser }); return headers; /* function arrayEqual(a, b) { if (a === b) return true; if (a == null || b == null) return false; if (a.length !== b.length) return false; for (var i = 0; i < a.length; ++i) { if (a.indexOf(b[i]) < 0) { return false; } } return true; } function isUnique(object, array) { return ( array.find(function(testObject) { return arrayEqual(object, testObject); }) == object ); } function patterns(edges) { let objects = []; for (let edge of edges) { let objectId = edge.source; let object = objects[objectId]; let label = edge.label; if (!object) { object = []; objects[objectId] = object; } object[label] = 1 + (object[label] || 0); } console.log(objects); objects = objects .filter(function(object) { return object; }) .map(function(object) { return Object.keys(object).sort(); }); console.log("after second filter", objects); return objects.filter(function(object) { return isUnique(object, objects); }); } return patterns(model.datasets[0].edges); */ }37:[{"id":"a50af9c19082ec0430d045a5e8d8b0f52571f059c34d7a73e08769d6330b11362c85e582c61c39ce57c4969c6c7f55b3075699c49e63181a4df67ecff8cb85bf","href":"https://static.observableusercontent.com/files/a50af9c19082ec0430d045a5e8d8b0f52571f059c34d7a73e08769d6330b11362c85e582c61c39ce57c4969c6c7f55b3075699c49e63181a4df67ecff8cb85bf","name":"fbfpt.sensors@1.svg","type":"image/svg+xml","lastModified":1614977693811,"size":21608},{"id":"ae2bfaf0962354dc00cae9ba2a4a1d9f0b073ee06fea0c7617f7fc6ee34245a8ba58c2f466f316348f1da396aaf6b4672baa5c738da6547e9755db3846348ddb","href":"https://static.observableusercontent.com/files/ae2bfaf0962354dc00cae9ba2a4a1d9f0b073ee06fea0c7617f7fc6ee34245a8ba58c2f466f316348f1da396aaf6b4672baa5c738da6547e9755db3846348ddb","name":"resource-workflows-model1@1.svg","type":"image/svg+xml","lastModified":1615810305922,"size":8202},{"id":"82d96afc1d7ad93d3f9186e445af6ced2c268ea6b69681a4a00a1a38979b5c7bafdfca683a12743c88cd7f470b472f303efd9c5ae863c008c5b06a1e1559a17e","href":"https://static.observableusercontent.com/files/82d96afc1d7ad93d3f9186e445af6ced2c268ea6b69681a4a00a1a38979b5c7bafdfca683a12743c88cd7f470b472f303efd9c5ae863c008c5b06a1e1559a17e","name":"service-wheel.svg","type":"image/svg+xml","lastModified":1615224420344,"size":119915},{"id":"306fe3dd485d843ec21e13f956df14faec7fa4c476d22375e62fcace4bc2c8aa9208c7a5f7cf94eb3d4b2325c15470b2c6070a32d11b351cca4ccaf624645c96","href":"https://static.observableusercontent.com/files/306fe3dd485d843ec21e13f956df14faec7fa4c476d22375e62fcace4bc2c8aa9208c7a5f7cf94eb3d4b2325c15470b2c6070a32d11b351cca4ccaf624645c96","name":"resource-workflows@2.svg","type":"image/svg+xml","lastModified":1614952306320,"size":8150}] 2a:{"title":"SOW-20210305 : Visible Workflows","cells":[{"id":0,"value":"md`# SOW-20210305 : Visible Workflows\nJames Anderson \nSemih Korkmaz \nIvan Labra \nMax-Gerd Retzlaff \n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":2,"value":"md`## Overview\nThe service can reify SPARQL queries and associate them with URL among an account's resources in the HTTP interface to be referenced internally in other SPARQL queries.\nThey mediate between various sources and destinations, they can\n\n- act as schema to define data entry interfaces,\n- operate as internal transformation instructions between repositories,\n- govern data import into a repository,\n- afford access to local and remote virtual sources,\n- fulfill the standard role as queries against repositories.\n${await FileAttachment(\"resource-workflows@2.svg\").image()}\n\nGiven that the views comprise all of the information required to specify the data and control flows, what is the best way to describe and control that data-flow process?\nIs it possible to represent everything with just the queries and use them to drive overview presentation and process control in addition to the individual operations, or must there be a dedicated representation and control process?\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":26,"value":"md`## Alternative Depictions\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":30,"value":"md`### graphviz\n\nIt is possible to produce a visual graph directly from the SPARQL queries by examining their algebra to identify their roles, as above.\nThat is, to differentiate TARQL-like processes, from update transformations, from federated interconnections, from simple document generation.\n\nUnfortunately, the depiction quickly reaches its legibility limits.\nA simple situation with perhaps a few dozen components is manageable.\n\n\n${await FileAttachment(\"fbfpt.sensors@1.svg\").image()}\n\nA more extensive collection, on the other hand yields incomprehendable visuals. \n(open in its own window:)\n\n

\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":42,"value":"md`### circos\n\nCircos offers a capacity to condense complex relationships into a compact image and to superimpose them into layered, parallel presentations.\n\nThis example illustrates how to display relations among several different sorts of entites in parallel and selectively.\n\n