30:T4e4,### Core Application Notebooks Three notebooks comprise the main entry points for the different types of system users: | Notebook | Audience | Purpose | |---|---|---|---|---| | [Filler](https://observablehq.com/@categorise/opensurvey?collection=@categorise/gesi-survey) | End Users / Those Answering Questions | Provides a view of the survey to be completed. Access is to survey fillers by way of authorized links. | | | | | [Designer](https://observablehq.com/@categorise/survey-designer?collection=@categorise/gesi-survey) | Survey Creators / Those asking questions | Provides an interface to create and edit survey content. | | | | | [Admin](https://observablehq.com/@categorise/survey-admin?collection=@categorise/gesi-survey) | Technical Owners / IT Administrators | Provides utilities for survey & user management, including definition of permissions / access controls and a means of deploying created surveys to CloudFront. | | | | Each application notebook can be accessed from the Observablehq domain, however the best way to view each page is through AWS S3/CloudFront, as doing so avoids potential style conflicts. Once deployed, each survey application is can be accessed and used without a dependency on Observable servers.31:T5e4,### Ecosystem Helper Notebooks Survey Slate is built using other resources contributed and shared openly by our team members and others. These open source notebooks are found within the Observable ecosystem. We'd like to highlight some of the bigger contributions: | Notebook | Purpose |---|---|---|---|---| | [AWS Helpers](https://observablehq.com/@tomlarkworthy/aws) | Helpers for accessing resources hosted on AWS based on the AWS SDK for JavaScript.| | | | | [Deploy Notebook to S3](https://observablehq.com/@tomlarkworthy/notebook-deploy-to-s3) | An architecture based on the AWS Helpers to help more easily deploys notebook to S3 (and invalidating CloudFront cache). | | [Notebook Secret](https://observablehq.com/@endpointservices/notebook-secret) | A means of encrypting notebook secrets. | [View literal](https://observablehq.com/@tomlarkworthy/view) | A helper function (and corresponding explanatory notes) intended to ease the burden of building complex UIs. | [UI development](https://observablehq.com/@tomlarkworthy/ui-development) | A tutorial on building complex UIs | [Jeremy Ashkenas' Inputs](https://observablehq.com/@jashkenas/inputs) | A set of UI components first published in 2018 and initially the 'go-to' means of adding interactive inputs to Observable notebooks. | [Observable's Inputs](https://observablehq.com/@observablehq/inputs) | Another set of UI components, released in 2021 and intended to help ensure greater inputs accessibility (for both humans and machines). 32:T42f,## Cloud Hosting Architecture Survey Slate uses AWS application hosting, data storage and user management. Service components are as follows: | AWS Service | purpose |---|---|---|---|---| | [S3](https://us-east-2.console.aws.amazon.com/console/home?region=us-east-2) | Cloud file storage. Separate buckets are used for surveys, responses and configuration. | [IAM](https://console.aws.amazon.com/iam/home?region=us-east-2) | User management and access policies. | [Cloud Front](https://console.aws.amazon.com/cloudfront/v3/home?region=us-east-2) | Content delivery (for application serving). The design concept aims for Cloud simplicity: - There is no database, only files and folder in S3 buckets which can be manipulated directly - There are no deployed compute resources. - Authorization is achieved with IAM resource policies on resource tags. Static configuration for resource layout is in the [component](https://observablehq.com/@categorise/survey-components) notebook in a cell named [config](https://observablehq.com/@categorise/survey-components#config). 33:T4c1,#### IAM User-Resource Policy A single IAM policy is attached to the 'Users' group which enforces constraints such as - Only users tagged as designers containing the value "survey1" can *read/write* S3 objects tagged with survey "survey1" in the private bucket - Only users tagged as filler containing the value "survey2" can *read* S3 objects tagged with survey "survey1" in the private bucket - Only users tagged as account containing the value "org1" can *read/write* S3 objects tagged with account "org1" in the confidential bucket The main trick to the IAM policy is using StringLike condition which will perform substring matching, which is the semantic we need to perform "containing" logic between a SCALAR and a LIST. See the following example. Note __*__ is wildcard matching. ```js "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::private/*", // Rule applies to access in the private bucket "Condition": { "StringLike": { // look for "project" SCALAR inside 'designer' LIST "aws:PrincipalTag/designer": "*\${s3:ExistingObjectTag/project}*" } } ```34:T92a,policy = ({ "Version": "2012-10-17", "Statement": [ { "Sid": "UserListOwnTags", "Effect": "Allow", "Action": [ "iam:GetUser", "iam:ListUserTags" ], "Resource": "arn:aws:iam::032151534090:user/${aws:username}" }, { "Sid": "DesignerReadSurvey", "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::private-mjgvubdpwmdipjsn/*", "Condition": { "StringLike": { "aws:PrincipalTag/designer": "*${s3:ExistingObjectTag/survey}*" } } }, { "Sid": "DesignerWriteSurvey", "Effect": "Allow", "Action": [ "s3:putObjectTagging", "s3:PutObject" ], "Resource": "arn:aws:s3:::private-mjgvubdpwmdipjsn/*", "Condition": { "StringLike": { "aws:PrincipalTag/designer": "*${s3:RequestObjectTag/survey}*" } } }, { "Sid": "FillerReadAccount", "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::confidential-bspqugxjstgxwjnt/*", "Condition": { "StringLike": { "aws:PrincipalTag/account": "*${s3:ExistingObjectTag/account}*" } } }, { "Sid": "FillerWriteAccount", "Effect": "Allow", "Action": [ "s3:putObjectTagging", "s3:PutObject" ], "Resource": "arn:aws:s3:::confidential-bspqugxjstgxwjnt/*", "Condition": { "StringLike": { "aws:PrincipalTag/account": "*${s3:RequestObjectTag/account}*" } } }, { "Sid": "FillerReadSurvey", "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::private-mjgvubdpwmdipjsn/*", "Condition": { "StringLike": { "aws:PrincipalTag/filler": "*${s3:ExistingObjectTag/survey}*" } } } ] })35:T6a3,### Magic Link Login Journey Here we outline the provisioning of an end user credentials that lead to a link which can complete surveys - Provision an IAM User - Generate access keys - Generate password - Encrypt access keys with password and place in public bucket `/credentials/.json` - Place password in private bucket `/password/.json` - Generate magic link `?username=#` The user is sent the magic link via email, which takes them directly to the survey filler app. Then - The application hashes the username - Looks up the encrypted IAM credential in `/credentials/.json` - Decrypts the payload with the password (found in URL hash) At this point the browser will be in possession of IAM access keys and can issue authenticated request to the AWS SDK. The first step will be querying for IAM tags, which will tell the user what surveys and accounts they have access to. From there they can read the relevant `**/settings.json` for survey/account metadata. **note:** - The password in the URL is after the hash to avoid transmission over the internet. - Staff members can be issued IAM access keys directly to avoid storing them encrypted on S3 - AWs limits access keys to two per user => we need to retain access to a users IAM keys indefinitely if we wish to reissue magic links without invalidating old links. - Storing passwords in S3 seems weird, but its not different to storing them in a DB (it's a KV store) - It is unusual to store passwords in plain text, but we need to be able generate new magic links on demand. Perhaps we could do better. At least s3 encrypts at-rest. 36:[] 2e:{"title":"Survey Slate | Technical Overview","cells":[{"id":0,"value":"# Survey Slate | Technical Overview\n\n_Documentation for getting started with your own survey application & orientation to application components._","pinned":false,"hidden":false,"mode":"md"},{"id":500,"value":"md`\n

👋 Welcome! This notebook is about **Survey Slate**—an [assemblage of Observable web-based notebooks](https://observablehq.com/collection/@categorise/survey-slate) allowing organizations to host custom surveys for end users on their own AWS infrastructure. ✨

\n\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":511,"value":"md`\n---\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":504,"value":"md`\nSurvey Slate allows organizations to host custom surveys for end users on their own AWS infrastructure. The Survey Slate system is comprised of an [assemblage of Observable web-based notebooks](#core-application-notebooks), each of which can be **forked** and customized **programmatically** using just a browser.\n\nThe main objectives in creating the Survey Slate application are:\n\n - Infinite customizability for future extensibility\n - Secure data isolation\n - Sign-in with link\n - BYO Cloud buckets for data compliance.\n\nThis notebook introduces the Survey Slate's architecture and walks through the process of deploying the application to help others implement and adapt it. Have question or wish to help us improve? Please feel welcome to open a comment or send a suggestion! And thank you for your time and interest!\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":329,"value":"toc()","pinned":false,"hidden":false,"mode":"ojs"},{"id":331,"value":"import {toc} from \"@bryangingechen/toc\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":101,"value":"md`## Application Software`","pinned":false,"hidden":false,"mode":"ojs"},{"id":21,"value":"$30","pinned":false,"hidden":false,"mode":"md"},{"id":60,"value":"### Application Customizations & Extension Notebooks\n\nTo extend and adapt application functionality and to customize application services programmatically, we utilize two additional notebooks:\n\n| Notebook | purpose \n|---|---|---|---|---|\n| [Components](https://observablehq.com/@categorise/survey-components) | Survey component library and global configuration| | | |\n| [Styling](https://observablehq.com/@categorise/gesi-styling) | Page layout and CSS for presenting a survey","pinned":false,"hidden":false,"mode":"md"},{"id":76,"value":"$31","pinned":false,"hidden":false,"mode":"md"},{"id":97,"value":"$32","pinned":false,"hidden":false,"mode":"md"},{"id":141,"value":"### S3 structure\n\nFor security and ease of comprehension, data are divided across three S3 buckets. Each bucket is prefixed by its security theme.\n\n| Bucket name prefix | purpose \n|---|---|---|---|---|\n| confidential | Contains end user data (survey responses).\n| private | Contains private org data (survey content and settings).\n| public | Contains externally accessible data (application software and encrypted end user credentials).\n\nWithin each bucket are several for-purposes directories, as described below.\n","pinned":false,"hidden":false,"mode":"md"},{"id":152,"value":"#### Confidential Bucket\n\nExternal customer data is confidential, and ringfenced within a dedicated bucket, organized as follows:\n\n| path | purpose \n|---|---|---|---|---|\n| `/accounts//settings.json` | Per-account data.\n| `/accounts//survey//answers_.json` | Survey answers.\n\nEach directory folder under `/accounts/survey/` represents a specific survey (set of questions), with an unlimited number of surveys being possible for each account. When an authorized account respondent (survey filler) responds or updates a survey, answers are stored in a time-stamped file. The per-account data file contains a pointer to the latest set of answers.\n","pinned":false,"hidden":false,"mode":"md"},{"id":167,"value":"#### Private Bucket\n\nThe private bucket is for internal organizational data\n\n- survey design data\n- passwords for authorized link access.\n\n| path | purpose \n|---|---|---|---|---|\n| `/surveys//settings.json` | Survey metadata\n| `/surveys//layout_.json` | Survey layout\n| `/surveys//questions_.json` | Survey questions\n| `/surveys//version_.json` | Survey versions\n| `/passwords/` | Password for magic URLs \n","pinned":false,"hidden":false,"mode":"md"},{"id":188,"value":"#### Public Bucket\n\nThe public bucket is for access on the open internet\n\n- exported app notebooks\n- password protected encrypted IAM credentials\n\n| path | purpose \n|---|---|---|---|---|\n| `/apps/designer/index.html` | Exported Designer notebook entrance point\n| `/apps/survey/index.html` | Exported Filler notebook entrance point\n| `/credentials/` | Encrypted IAM credentials for AWS access\n","pinned":false,"hidden":false,"mode":"md"},{"id":201,"value":"### IAM and security\n\nBuckets contain sensitive data, access to the S3 is protected with AWS IAM policies. We achieve Attribute Based Access Control by extensive use of Resource Tags.\n\n- relevant AWS article [Controlling access to AWS resources using tags](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_tags.html).\n\nOrganization staff and end users are grouped together under one IAM user group called \"Users\". However, what particular objects in s3 can be access depends on \n\n- Tags on the user's Principle (see [Controlling access to and for IAM users and roles using tags](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_iam-tags.html))\n- Tags on the object being access in S3 (see [Controlling access to AWS resources using tags](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_tags.html))\n\nThus, a single IAM user *may* be a designer for one survey, and a survey filler for another.\n\n","pinned":false,"hidden":false,"mode":"md"},{"id":214,"value":"#### IAM User Groups\n\nWe use two IAM user groups\n\n- Admins (technical staff like developers who access the AWS console)\n- Users (All other users, including Survey Slate admins, designers and fillers)","pinned":false,"hidden":false,"mode":"md"},{"id":220,"value":"#### Principle Tags\n\nIAM users who are members of the group \"Users\" are tagged to represent data access\n\n| Principle tag key | value\n|---|---|---|---|---|\n| `designer` | List of surveys the user is authorized to modify\n| `account` | List of organizations the user is part of\n| `filler` | List of surveys the user can respond to\n\nWe make extensive use of embedded lists in values, using `\" \"` to delimited entries.*\n\n_* In retrospect, we would use `\"|\"`—including on the start and end of the value—to simplify the AWS policy expression._\n","pinned":false,"hidden":false,"mode":"md"},{"id":238,"value":"#### S3 Object Tags\n\nObjects in S3 are tagged to denote who has read/write access to them.\n\n| Bucket | Resource tag key | value\n|---|---|---|---|---|\n| `private` | `survey` | The survey this object belongs to\n| `confidential` | `account` | The organization this object belongs to\n","pinned":false,"hidden":false,"mode":"md"},{"id":251,"value":"$33","pinned":false,"hidden":false,"mode":"md"},{"id":274,"value":"#### IAM User Capability Self-Discovery\n\nUsers are granted permission to read their own tags so they can discover what they have access too.\n\n```js\n \"Effect\": \"Allow\",\n \"Action\": [\n \"iam:GetUser\",\n \"iam:ListUserTags\"\n ],\n \"Resource\": \"arn:aws:iam::032151534090:user/\\${aws:username}\"\n }\n```","pinned":false,"hidden":false,"mode":"md"},{"id":287,"value":"md`### Reference Policy`","pinned":false,"hidden":false,"mode":"ojs"},{"id":258,"value":"$34","pinned":false,"hidden":false,"mode":"ojs"},{"id":292,"value":"$35","pinned":false,"hidden":false,"mode":"md"},{"id":312,"value":"### CloudFront Application Hosting\n\nThe applications are implemented as Notebooks for ease of development. For deployment, the notebooks are exported as a tarball, and placed\nin S3. AWS's CDN CloudFront then severs the content over HTTPS from the S3 bucket..\n\nThe notebook [deploy notebook to s3](https://observablehq.com/@tomlarkworthy/notebook-deploy-to-s3) is used to provide a top level HTML framing, over Observable's [export notebook](https://observablehq.com/@observablehq/downloading-and-embedding-notebooks#cell-291) tarball.\n","pinned":false,"hidden":false,"mode":"md"},{"id":326,"value":"## Installing on your own AWS account\n\nOur notebooks are attached to a private AWS account with no public access. This section will explain how to move these applications onto your own AWS infrastructure. We have no spent much engineering time on this story yet, but we welcome external contributions to streamline it.","pinned":false,"hidden":false,"mode":"md"},{"id":348,"value":"### Provision AWS resources\n\nYou need three buckets, a \"user\" IAM group with an attached \"user\" IAM policy.\n\nYou will need to update the IAM policy to reference your AWS account number and your chosen bucket names.\n\nCloudFront is not so important and can be done later.","pinned":false,"hidden":false,"mode":"md"},{"id":340,"value":"### Fork notebooks and update dependencies\n\nFork the core and support notebooks. Ensure the forked core notebook dependencies point to the new support notebooks.","pinned":false,"hidden":false,"mode":"md"},{"id":357,"value":"### Set AWS configuration\n\nUpdate the configuration found in the forked notebook of [survey-components](https://observablehq.com/@categorise/survey-components)\n~~~js\nconfig = ({\n // All artifacts in the PUBLIC bucket are on the internet\n PUBLIC_BUCKET: \"public-publicwrfxcsvqwpcgcrpx\",\n // The private bucket requires credentials to access, should be for system wide things like user lists\n PRIVATE_BUCKET: \"private-mjgvubdpwmdipjsn\",\n // The confidential bucket is for customer data, it is encrypted and access should be minimized.\n CONFIDENTIAL_BUCKET: \"confidential-bspqugxjstgxwjnt\",\n // The URL to redirect users to\n FILLER_APP_URL: \"https://observablehq.com/@categorise/opensurvey\",\n // Cloud Front distribution ID\n CLOUD_FRONT_DISTRIBUTION_ID: 'EG13QGKCG6LI9',\n // URL that hosts applications\n CLOUD_FRONT_URL: 'https://d3f26ej57oauer.cloudfront.net',\n})\n~~~","pinned":false,"hidden":false,"mode":"md"},{"id":365,"value":"### Create a Admin access key\n\nCreate access keys for the person who should be able to create users. Those credentials will be be used with the [admin app](https://observablehq.com/@categorise/survey-admin).","pinned":false,"hidden":false,"mode":"md"},{"id":382,"value":"### Create a test survey\n\nUsing the [admin app](https://observablehq.com/@categorise/survey-admin) and the admin credentials. \n\n- \"Create New Survey\"\n","pinned":false,"hidden":false,"mode":"md"},{"id":378,"value":"### Provision a test user with internal and external access\n\n- \"Create a new User\"\n- \"Choose a user to configure\" -> select your test user\n- \"Create an external access code (suitable for Survey Fillers)\"\n- \"Access links\" -> \"Generate Access link\" and **_make a note_** of the access link\n- \"Create an internal access key (suitable for designers)\" and **_make a note_** of the access key\n","pinned":false,"hidden":false,"mode":"md"},{"id":398,"value":"### Provision a test accounts\n\n- \"Create a new account\"\n","pinned":false,"hidden":false,"mode":"md"},{"id":396,"value":"### Grant the test user design and filler permissions to your test project and test account.\n\nFirst select your test user\n- \"Choose a user to configure\" -> select your test user\n\n\nThen make them a designer\n- \"Survey Designer Access Tags \" -> \"Select project for operation\" -> select your test survey\n- \"add to as designer\"\n\nGrant them filler access\n- \"Survey Filler Access Tags \" -> \"Select project for operation\" -> select your test survey\n- \"add to as filler\"\n\nGrant them access to the test organization\n- \"Survey Filler Account Tags\" -> \"select account for operation\" -> select your test account\n- \"Allow to respond on behalf of \"","pinned":false,"hidden":false,"mode":"md"},{"id":424,"value":"### Test your user\n\nYou should be able to use the internal access key to use the forked [Designer app](https://observablehq.com/@categorise/survey-designer?collection=@categorise/gesi-survey). You should be able to use the external access link to use the forked [Filler app](https://observablehq.com/@categorise/opensurvey?collection=@categorise/gesi-survey). You can tell if the access keys are working if you see the account/project/survey drop downs get populated. \n\n⚠️ Check the Console logs if you encounter problems (option + command + i usually Chrome)\n","pinned":false,"hidden":false,"mode":"md"},{"id":492,"value":"md`\n---\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":496,"value":"md`\nA tiny helper for page jumping, contributed by [@mootari](https://observablehq.com/@mootari) below (undefined).\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":490,"value":"{\n const onclick = e => {\n if(e.target.tagName !== 'A') return;\n const href = e.target.getAttribute('href');\n if(href.startsWith('#')) {\n const t = document.querySelector(href);\n if(!t) return;\n e.preventDefault();\n t.scrollIntoView();\n }\n };\n document.body.addEventListener('click', onclick);\n invalidation.then(() => document.body.removeEventListener('click', onclick));\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":527,"value":"---","pinned":false,"hidden":false,"mode":"md"},{"id":529,"value":"import { substratum } from \"@categorise/substratum\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":531,"value":"substratum({ invalidation })","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W36","stdlib":"1","titling":"auto"}