2f:[{"id":"18d929901383fce8dfffd144fc08ad9d998d2a7a0ce5473fc70a1e5622a8c9e8757a599ac484a5a20e38f368db07b68c70b2abd6a82f85e08b0e2dbdeae84609","href":"https://static.observableusercontent.com/files/18d929901383fce8dfffd144fc08ad9d998d2a7a0ce5473fc70a1e5622a8c9e8757a599ac484a5a20e38f368db07b68c70b2abd6a82f85e08b0e2dbdeae84609","name":"Editor_Hints_Blog.png","type":"image/png","lastModified":1674633019568.351,"size":86087},{"id":"d1c0930c41f7854d2d7c5f666262c4356310d43d1e8557fa2d48907912ef4272b36fd6abea6b53b8f80dbd0daf277c8a03ede36b36cfe3dce79da92bea3f5403","href":"https://static.observableusercontent.com/files/d1c0930c41f7854d2d7c5f666262c4356310d43d1e8557fa2d48907912ef4272b36fd6abea6b53b8f80dbd0daf277c8a03ede36b36cfe3dce79da92bea3f5403","name":"image@1.png","type":"image/png","lastModified":1674633019568.351,"size":119505},{"id":"be852d9e294b982f8f4938aad1eab0684e3ecf3900dacbeefce5ef0a8aaaff83a85ed50e95a65b2704c67958d8ec0951281de7ba0f5cc0869e0aeaa2136c0e3b","href":"https://static.observableusercontent.com/files/be852d9e294b982f8f4938aad1eab0684e3ecf3900dacbeefce5ef0a8aaaff83a85ed50e95a65b2704c67958d8ec0951281de7ba0f5cc0869e0aeaa2136c0e3b","name":"image@4.png","type":"image/png","lastModified":1674633019568.351,"size":231356}] 2d:{"title":"DRAFT","cells":[{"id":135,"value":"# DRAFT","pinned":false,"hidden":false,"mode":"md"},{"id":299,"value":"The editor hints found in popular code editors such as VSCode (_e.g._, autocomplete and embedded documentation) have become indispensable for developers. So, we set out to implement similar functionality within Observable. \n\nWe learned that the architecture that powers VSCode is modular and that the “brains” of VSCode can be used outside of VSCode, even in a browser. After a few twists and turns, we were able to implement editor hints within Observable by integrating with the TypeScript Language Server. \n\nThis is how we did it.","pinned":false,"hidden":false,"mode":"md"},{"id":164,"value":"## An integration hidden in plain sight\n\nCode editors often use the [TypeScript Language Server](https://github.com/typescript-language-server/typescript-language-server) (`tsserver`) to provide editor hints for TypeScript as well as JavaScript. For JavaScript, completions are inferred either from JavaScript source found in the project or from any d.ts files that have been loaded into the project (_e.g._, all of the node or browser APIs). \n\n`tsserver` also consumes any JSDoc annotations it finds in order to display signature help descriptions, as well as any JSDoc _type_ annotations to provide autocompletions, _etc._ \n\nHere is an example from the TypeScript Playground:","pinned":false,"hidden":false,"mode":"md"},{"id":149,"value":"

Signature help as well as autocompletions via JSDoc. Try it in the TypeScript Playground.

","pinned":false,"hidden":false,"mode":"html"},{"id":204,"value":"The question we had when starting this project was whether we could get these editor hints into Observable, which runs in the browser and uses [CodeMirror](https://codemirror.net/) to as an editor. VSCode is powered by Microsoft’s open-source editor called [Monaco](https://microsoft.github.io/monaco-editor/), so it wasn’t clear how we could integrate with CodeMirror, if at all. Luckily, the Monaco editor is implemented in a modular way, with a clear separation between the editor and the language services.","pinned":false,"hidden":false,"mode":"md"},{"id":206,"value":"The Monaco editor doesn’t do any of the heavy lifting for the editor hints, but is more of the presentation layer that relies on the various language services (`tsserver` being one of them) to get the editor hint information. In theory this means we could either host a TS Language Server that the browser can query, or we could try to run `tsserver` in the browser to remove the round trip over the internet.","pinned":false,"hidden":false,"mode":"md"},{"id":167,"value":"We quickly discovered that `tsserver` has a dependency on the file system in order to read from and write to the various files in a TypeScript (or JavaScript) project, which meant there wouldn’t be a way to run `tsserver` in the browser. It wasn’t until we happened upon this [incredibly helpful CodeMirror forum thread](https://discuss.codemirror.net/t/codemirror-6-and-typescript-lsp/3398/3) that the path forward was clear. The answer was hidden in plain sight: do what the TypeScript Playground does and use [`@typescript/vfs`](https://www.npmjs.com/package/@typescript/vfs) rather than `tsserver`.","pinned":false,"hidden":false,"mode":"md"},{"id":170,"value":"## A little help from Sid\n\nUntil we found the above [CodeMirror forum thread](https://discuss.codemirror.net/t/codemirror-6-and-typescript-lsp/3398/3) and especially the responses by [Sid](https://discuss.codemirror.net/u/madebysid/summary), it hadn’t occurred to us that the [TypeScript Playground](https://www.typescriptlang.org/play) is actually already doing the thing we’re trying to do: run `tsserver` in the browser. We assumed (without ever actually checking) that it must be communicating over the internet with a hosted `tsserver` somewhere. \n\nSid invested a lot of time in that thread outlining how he got the TypeScript language service running in the browser and integrated with CodeMirror. [`@typescript/vfs`](https://www.npmjs.com/package/@typescript/vfs) was developed by Microsoft in order to enable implementation of the TypeScript Playground by removing `tsserver`’s dependency on the file system. (VFS here stands for Virtual File System.)","pinned":false,"hidden":false,"mode":"md"},{"id":214,"value":"Here we express our gratitude for Sid and Microsoft for paving the way. All that was left for us was to connect everything together. Let's do it now!","pinned":false,"hidden":false,"mode":"md"},{"id":327,"value":"## Try it yourself\n\nFirst, we import `typescript` and `@typescript/vfs`.","pinned":false,"hidden":false,"mode":"md"},{"id":670,"value":"kysely = import(\"https://cdn.jsdelivr.net/npm/kysely/dist/esm/index.js\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":330,"value":"vfs = import(\"https://cdn.jsdelivr.net/npm/@typescript/vfs@1.4.0/+esm\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":332,"value":"ts = (await import(\"https://cdn.jsdelivr.net/npm/typescript@4.8.4/+esm\")).default","pinned":true,"hidden":false,"mode":"ojs"},{"id":687,"value":"Kysely = kysely.Kysely","pinned":true,"hidden":false,"mode":"ojs"},{"id":672,"value":"DummyDriver = kysely.DummyDriver","pinned":true,"hidden":false,"mode":"ojs"},{"id":674,"value":"SqliteAdapter = kysely.SqliteAdapter","pinned":true,"hidden":false,"mode":"ojs"},{"id":676,"value":"SqliteIntrospector = kysely.SqliteIntrospector","pinned":true,"hidden":false,"mode":"ojs"},{"id":678,"value":"SqliteQueryCompiler = kysely.SqliteQueryCompiler","pinned":true,"hidden":false,"mode":"ojs"},{"id":334,"value":"Then we initialize a \"Virtual TypeScript Environment\". \n\nThis uses the `createDefaultMapFromCDN` function which will load all of the browser `lib.*.d.ts` files, which is how we have types and documentation for browser APIs like `document.querySelector()`.","pinned":false,"hidden":false,"mode":"md"},{"id":336,"value":"tsEnv = {\n storage;\n const {createSystem, createVirtualTypeScriptEnvironment, createDefaultMapFromCDN} = vfs;\n const fsMap = await createDefaultMapFromCDN({target: ts.ScriptTarget.ES2021}, ts.version, true, ts);\n const system = createSystem(fsMap);\n const tsEnv = createVirtualTypeScriptEnvironment(system, [], ts, {allowJs: true});\n tsEnv.createFile(\"/index.js\", \" \"); // This is where our code will go. Note: can’t be empty 😅\n return tsEnv;\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":339,"value":"As long as we keep the virtual TypeScript environment in sync with our editor, we will be able to leverage all of the smarts that the language server provides. \n\nOur editor in this notebook will be an html textarea (with a little extra to emit changes to the cursor offset as well). We will also query the language service for any auto completions we can show.","pinned":false,"hidden":false,"mode":"md"},{"id":360,"value":"completions = {\n tsEnv.updateFile(\"/index.js\", editor1.code || \" \");\n return tsEnv.languageService.getCompletionsAtPosition(\"/index.js\", editor1.offset, {});\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":347,"value":"viewof editor1 = InputWithSelection(\n \"interface Person {\\\n id: Generated\\\n first_name: string\\\n last_name: string | null\\\n}\\n\\\ninterface Database {\\\n person: Person\\\n}\\\n\"\n)","pinned":false,"hidden":false,"mode":"ojs"},{"id":357,"value":"Completions at position ${editor1.offset}: ${completions ? completions.entries.slice(0, 5).map(({name}) => name).join(\", \") : \"no completions\"}","pinned":false,"hidden":false,"mode":"html"},{"id":692,"value":"sql = db.selectFrom(\"person\").select(\"id\").compile()","pinned":true,"hidden":false,"mode":"ojs"},{"id":680,"value":"db = new Kysely({\n dialect: {\n createAdapter() {\n return new SqliteAdapter();\n },\n createDriver() {\n return new DummyDriver();\n },\n // createIntrospector(db: Kysely) {\n // return new SqliteIntrospector(db)\n // },\n createQueryCompiler() {\n return new SqliteQueryCompiler();\n }\n }\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":375,"value":"We can also get signature help items.","pinned":false,"hidden":false,"mode":"md"},{"id":379,"value":"help = {\n tsEnv.updateFile(\"/index.js\", editor2.code || \" \");\n return tsEnv.languageService.getSignatureHelpItems(\"/index.js\", editor2.offset, {});\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":385,"value":"viewof editor2 = InputWithSelection(\"document.getElementById(\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":389,"value":"signature = displayHelpItems(help, editor2.offset);","pinned":false,"hidden":false,"mode":"ojs"},{"id":176,"value":"We represent each cell in a notebook as a separate file in the VFS, and keep the TypeScript virtual environment in sync with changes made in the CodeMirror cell editors. \n\nThen, we can query the language service for signature help and autocompletions at any position in the CodeMirror editor. What remains is simply the UI work around integrating these autocompletions and signature help items with CodeMirror. \n\nFortunately, CodeMirror has very robust frameworks for both [autocompletions](https://codemirror.net/examples/autocompletion/) as well as [tooltips](https://codemirror.net/examples/tooltip/).","pinned":false,"hidden":false,"mode":"md"},{"id":220,"value":"Next, we moved the virtual TypeScript environment to a WebWorker. That way, we could then take advantage of [Webpack 5’s automatic code-splitting for WebWorkers](https://webpack.js.org/guides/web-workers/) to ensure that the download of the virtual TypeScript environment and the download of the Observable application would be done separately. \n\nThis also alleviated concerns about performance because [WebWorkers are run in a separate thread](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers), so any heavy lifting that the TypeScript language service is doing will not block the main thread and the UI will still be responsive.","pinned":false,"hidden":false,"mode":"md"},{"id":224,"value":"\nNow, when we use native browser APIs in a cell, we get the autocompletions and signature help:\n","pinned":false,"hidden":false,"mode":"md"},{"id":143,"value":"

","pinned":false,"hidden":false,"mode":"html"},{"id":593,"value":"## What's next: transpile and more\n\nThere are still some challenges we’re working through.","pinned":false,"hidden":false,"mode":"md"},{"id":226,"value":"### Transpile\n\nThe first issue we had to face is the fact that [Observable isn’t JavaScript](https://observablehq.com/@observablehq/observables-not-javascript). There are some tweaks and additions we’ve made to JavaScript in order to facilitate the reactivity and express certain functionality in Observable. \n\nIn order for the TypeScript language service to be able to make sense of a given cell, we need to _transpile_ that cell into something that is valid JS first, and maintain a Source Map that will allow us to map from a position in the CodeMirror editor to a position in the transpiled JavaScript file that is in the TypeScript virtual environment.","pinned":false,"hidden":false,"mode":"md"},{"id":228,"value":"### Cross-cell inference improvements\n\nAnother issue is that the [Observable Runtime](https://github.com/observablehq/runtime) does really nice things like ensure that a Promise is resolved before evaluating downstream cells. Unfortunately, we don’t have a way yet to make TypeScript aware of all of these transformations, so there are some instances where cross-cell inference is incorrect. \n\nSo, if you reference a cell that is defined as a Promise in another cell, you will get completions for a promise, not what the Promise resolves to. We will be improving cross-cell inference over time.\n","pinned":false,"hidden":false,"mode":"md"},{"id":141,"value":"### D.ts file views\nAnother issue is that within an IDE, if you see a named TypeScript interface in, say, a Signature help tooltip, there is a way to drill down on that interface and see what its definition is. This is not yet implemented in Observable, so sometimes you will see a reference to a named interface or type, but not have a way to “drill down” and view that interface’s definition. We intend to improve this experience so you can view the d.ts files from the virtual TypeScript environment.\n","pinned":false,"hidden":false,"mode":"md"},{"id":230,"value":"### Our Typescript roadmap\n\nFinally, some things we would like to do in the future are:\n\n- Observable users should be able to use JSDoc annotations on cell functions to provide Signature help and autocompletions for their libraries.\n- We want to use TypeScript’s [Automatic Type Acquisition library](https://www.npmjs.com/package/@typescript/ata) to fetch the types (if they exist) for any user-required (or imported) modules.\n- We also want to provide editor hints for all of Observable’s standard Library.","pinned":false,"hidden":false,"mode":"md"},{"id":644,"value":"banner(\n \"Interested in learning more about collaborating with data?\",\n \"Sign up for free\",\n \"https://observablehq.com/product\"\n)","pinned":false,"hidden":false,"mode":"ojs"},{"id":646,"value":"import {banner} from \"@observablehq/brand-assets\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":343,"value":"---\n\n## Appendix","pinned":false,"hidden":false,"mode":"md"},{"id":345,"value":"InputWithSelection = (initialValue) => {\n let value = initialValue;\n const input = htl.html`${value}`;\n\n function onChange() {\n form.value = {\n code: input.value,\n offset:\n document.activeElement === input\n ? input.selectionStart\n : input.value.length\n };\n \n form.dispatchEvent(new CustomEvent(\"input\", { bubbles: true }));\n }\n\n const form = htl.html``;\n form.value = { code: initialValue, offset: initialValue.length };\n return form;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":397,"value":"function displayHelpItem(item) {\n const signaturePrefix = ts.displayPartsToString(item.prefixDisplayParts);\n const signatureParameters = item.parameters\n .map((param) => ts.displayPartsToString(param.displayParts))\n .join(\", \");\n const signatureSuffix = ts.displayPartsToString(item.suffixDisplayParts);\n return htl.html`

${signaturePrefix}${signatureParameters}${signatureSuffix}

${ts.displayPartsToString(item.documentation)}

`;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":398,"value":"function displayHelpItems(help, offset) {\n return htl.html`

\n Signature help at position ${offset}:

\n      ${\n        !help\n          ? \"no help items\"\n          : displayHelpItem(help.items[help.selectedItemIndex])\n      }\n

`;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":666,"value":"storage = {\n Object.defineProperty(window, \"localStorage\", {value: localStorage});\n return localStorage;\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":665,"value":"import {localStorage} from \"@mbostock/safe-local-storage\"","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W2f","stdlib":"1","titling":"auto"}