wip

README.md

@@ -1,21 +1,89 @@
-# bun-react-tailwind-template
+# Shrimp Parser - Development Context
 
-To install dependencies:
+## Overview
 
-```bash
+Building a command-line language parser using Lezer (CodeMirror's parser system) with TypeScript. The goal is to create a prototype that can parse commands with arguments, similar to shell syntax, with inline hints for autocompletion.
-bun install
+
+## Current Architecture
+
+### Grammar Structure (`shrimp.grammar`)
+
+- **Commands**: Can be complete (`Command`) or partial (`CommandPartial`) for autocomplete
+- **Arguments**: Positional or named (with `name=value` syntax)
+- **Key Challenge**: Handling arbitrary text (like file paths) as arguments without conflicting with operators/keywords
+
+### Tokenizer Setup (`tokenizers.ts`)
+
+- **Main tokenizer**: Returns `Command`, `CommandPartial`, or `Identifier` based on context
+- **Command matching**: Uses `matchCommand()` to check against available commands
+- **Context-aware**: Uses `stack.canShift()` to return appropriate token based on parse position
+- **Issue**: Second occurrence of command name (e.g., `tail tail`) should be `Identifier` not `Command`
+
+### Key Design Decisions
+
+1. **External tokenizers over regular tokens** for commands to enable:
+
+   - Dynamic command list (can change at runtime)
+   - Partial matching for autocomplete
+   - Context-aware tokenization
+
+2. **Virtual semicolons** for statement boundaries:
+
+   - Using `insertSemicolon` external tokenizer
+   - Inserts at newlines/EOF to keep parser "inside" CommandCall
+   - Prevents `tail t` from parsing as two separate commands
+
+3. **UnquotedArg token** for paths/arbitrary text:
+   - Accepts anything except whitespace/parens/equals
+   - Only valid in command argument context
+   - Avoids conflicts with operators elsewhere
+
+### Current Problems
+
+1. **Parser completes CommandCall too early**
+
+   - After `tail `, cursor shows position in `Program` not `CommandCall`
+   - Makes hint system harder to implement
+
+2. **Command token in wrong context**
+
+   - `tail tail` - second "tail" returns `Command` token but should be `Identifier`
+   - Need better context checking in tokenizer
+
+3. **Inline hints need to be smarter**
+   - Must look backward to find command context
+   - Handle cases where parser has "completed" the command
+
+### Test Infrastructure
+
+- Custom test matchers: `toMatchTree`, `toEvaluateTo`
+- Command source injection for testing: `setCommandSource()`
+- Tests in `shrimp.test.ts`
+
+### File Structure
+
+```
+src/parser/
+  shrimp.grammar     - Lezer grammar definition
+  tokenizers.ts      - External tokenizers
+  shrimp.ts         - Generated parser
+
+src/editor/
+  commands.ts        - Command definitions
+  plugins/
+    inlineHints.tsx  - Autocomplete hint UI
 ```
 
-To start a development server:
+## Next Steps
 
-```bash
+1. Fix tokenizer context checking with `stack.canShift()`
-bun dev
+2. Improve hint detection for "after command with space" case
-```
+3. Consider if grammar structure changes would help
 
-To run for production:
+## Key Concepts to Remember
 
-```bash
+- Lezer is LR parser - builds tree bottom-up
-bun start
+- External tokenizers run at each position
-```
+- `@skip { space }` makes whitespace invisible to parser
-
+- Token precedence matters for overlap resolution
-This project was created using `bun init` in bun v1.2.22. [Bun](https://bun.com) is a fast all-in-one JavaScript runtime.
+- `stack.canShift(tokenId)` checks if token is valid at current position

src/editor/commands.ts

@@ -2,6 +2,7 @@ export type CommandShape = {
   command: string
   description?: string
   args: ArgShape[]
+  execute: string | ((...args: any[]) => any)
 }
 
 type ArgShape<T extends keyof ArgTypeMap = keyof ArgTypeMap> =
@@ -29,6 +30,7 @@ const commandShapes: CommandShape[] = [
   {
     command: 'ls',
     description: 'List the contents of a directory',
+    execute: './commands/ls.ts',
     args: [
       { name: 'path', type: 'string', description: 'The path to list' },
       { name: 'all', type: 'boolean', description: 'Show hidden files', default: false },
@@ -46,12 +48,14 @@ const commandShapes: CommandShape[] = [
   {
     command: 'cd',
     description: 'Change the current working directory',
+    execute: './commands/cd.ts',
     args: [{ name: 'path', type: 'string', description: 'The path to change to' }],
   },
 
   {
     command: 'cp',
     description: 'Copy files or directories',
+    execute: './commands/cp.ts',
     args: [
       { name: 'source', type: 'string', description: 'Source file or directory' },
       { name: 'destination', type: 'string', description: 'Destination path' },
@@ -63,6 +67,7 @@ const commandShapes: CommandShape[] = [
   {
     command: 'mv',
     description: 'Move files or directories',
+    execute: './commands/mv.ts',
     args: [
       { name: 'source', type: 'string', description: 'Source file or directory' },
       { name: 'destination', type: 'string', description: 'Destination path' },
@@ -73,6 +78,7 @@ const commandShapes: CommandShape[] = [
   {
     command: 'rm',
     description: 'Remove files or directories',
+    execute: './commands/rm.ts',
     args: [
       { name: 'path', type: 'string', description: 'Path to remove' },
       { name: 'recursive', type: 'boolean', description: 'Remove recursively', default: false },
@@ -84,6 +90,7 @@ const commandShapes: CommandShape[] = [
   {
     command: 'mkdir',
     description: 'Create directories',
+    execute: './commands/mkdir.ts',
     args: [
       { name: 'path', type: 'string', description: 'Directory path to create' },
       { name: 'verbose', type: 'boolean', description: 'Verbose output', default: false },
@@ -93,6 +100,7 @@ const commandShapes: CommandShape[] = [
   {
     command: 'touch',
     description: 'Create empty files or update timestamps',
+    execute: './commands/touch.ts',
     args: [
       { name: 'path', type: 'string', description: 'File path to touch' },
       { name: 'access', type: 'boolean', description: 'Update access time only', default: false },
@@ -108,6 +116,7 @@ const commandShapes: CommandShape[] = [
   {
     command: 'echo',
     description: 'Display a string',
+    execute: './commands/echo.ts',
     args: [
       { name: 'text', type: 'string', description: 'Text to display' },
       { name: 'no-newline', type: 'boolean', description: "Don't append newline", default: false },
@@ -117,6 +126,7 @@ const commandShapes: CommandShape[] = [
   {
     command: 'cat',
     description: 'Display file contents',
+    execute: './commands/cat.ts',
     args: [
       { name: 'path', type: 'string', description: 'File to display' },
       { name: 'numbered', type: 'boolean', description: 'Show line numbers', default: false },
@@ -126,6 +136,7 @@ const commandShapes: CommandShape[] = [
   {
     command: 'head',
     description: 'Show first lines of input',
+    execute: './commands/head.ts',
     args: [
       { name: 'path', type: 'string', description: 'File to read from' },
       { name: 'lines', type: 'number', description: 'Number of lines', default: 10 },
@@ -135,6 +146,7 @@ const commandShapes: CommandShape[] = [
   {
     command: 'tail',
     description: 'Show last lines of input',
+    execute: './commands/tail.ts',
     args: [
       { name: 'path', type: 'string', description: 'File to read from' },
       { name: 'lines', type: 'number', description: 'Number of lines', default: 10 },
@@ -145,6 +157,7 @@ const commandShapes: CommandShape[] = [
   {
     command: 'grep',
     description: 'Search for patterns in text',
+    execute: './commands/grep.ts',
     args: [
       { name: 'pattern', type: 'string', description: 'Pattern to search for' },
       {
@@ -161,6 +174,7 @@ const commandShapes: CommandShape[] = [
   {
     command: 'sort',
     description: 'Sort input',
+    execute: './commands/sort.ts',
     args: [
       { name: 'reverse', type: 'boolean', description: 'Sort in reverse order', default: false },
       {
@@ -176,6 +190,7 @@ const commandShapes: CommandShape[] = [
   {
     command: 'uniq',
     description: 'Filter out repeated lines',
+    execute: './commands/uniq.ts',
     args: [
       { name: 'count', type: 'boolean', description: 'Show count of occurrences', default: false },
       {
@@ -191,24 +206,28 @@ const commandShapes: CommandShape[] = [
   {
     command: 'select',
     description: 'Select specific columns from data',
+    execute: './commands/select.ts',
     args: [{ name: 'columns', type: 'string', description: 'Columns to select' }],
   },
 
   {
     command: 'where',
     description: 'Filter data based on conditions',
+    execute: './commands/where.ts',
     args: [{ name: 'condition', type: 'string', description: 'Filter condition' }],
   },
 
   {
     command: 'group-by',
     description: 'Group data by column values',
+    execute: './commands/group-by.ts',
     args: [{ name: 'column', type: 'string', description: 'Column to group by' }],
   },
 
   {
     command: 'ps',
     description: 'List running processes',
+    execute: './commands/ps.ts',
     args: [
       { name: 'long', type: 'boolean', description: 'Show detailed information', default: false },
     ],
@@ -217,12 +236,14 @@ const commandShapes: CommandShape[] = [
   {
     command: 'sys',
     description: 'Show system information',
+    execute: './commands/sys.ts',
     args: [],
   },
 
   {
     command: 'which',
     description: 'Find the location of a command',
+    execute: './commands/which.ts',
     args: [
       { name: 'command', type: 'string', description: 'Command to locate' },
       { name: 'all', type: 'boolean', description: 'Show all matches', default: false },

src/editor/editor.tsx

@@ -41,7 +41,7 @@ export const Editor = () => {
               shrimpLanguage(),
               shrimpHighlighting,
               shrimpErrors,
-              inlineHints,
+              // inlineHints,
               debugTags,
             ],
           })

src/evaluator/evaluator.test.ts

@@ -1,8 +1,83 @@
+import { resetCommandSource, setCommandSource, type CommandShape } from '#editor/commands'
 import { expect, test } from 'bun:test'
 
-test('parses simple assignments', () => {
+test('number literal', () => {
-  expect('number = 5').toEvaluateTo(5)
+  expect('42').toEvaluateTo(42)
-  expect('number = -5.3').toEvaluateTo(-5.3)
-  expect(`string = 'abc'`).toEvaluateTo('abc')
-  expect('boolean = true').toEvaluateTo(true)
 })
+
+test('negative number', () => {
+  expect('-5').toEvaluateTo(-5)
+})
+
+test('string literal', () => {
+  expect(`'hello'`).toEvaluateTo('hello')
+})
+
+test('boolean true', () => {
+  expect('true').toEvaluateTo(true)
+})
+
+test('boolean false', () => {
+  expect('false').toEvaluateTo(false)
+})
+
+test('addition', () => {
+  expect('2 + 3').toEvaluateTo(5)
+})
+
+test('subtraction', () => {
+  expect('10 - 4').toEvaluateTo(6)
+})
+
+test('multiplication', () => {
+  expect('3 * 4').toEvaluateTo(12)
+})
+
+test('division', () => {
+  expect('15 / 3').toEvaluateTo(5)
+})
+
+test('assign number', () => {
+  expect('x = 5').toEvaluateTo(5)
+})
+
+test('emoji assignment to number', () => {
+  expect('💎 = 5').toEvaluateTo(5)
+})
+
+test('assign string', () => {
+  expect(`name = 'Alice'`).toEvaluateTo('Alice')
+})
+
+test('assign expression', () => {
+  expect('sum = 2 + 3').toEvaluateTo(5)
+})
+
+test('parentheses', () => {
+  expect('(2 + 3) * 4').toEvaluateTo(20)
+})
+
+test('simple command', () => {
+  const commands: CommandShape[] = [
+    {
+      command: 'echo',
+      args: [{ name: 'text', type: 'string' }],
+      execute: (text: string) => text,
+    },
+  ]
+
+  withCommands(commands, () => {
+    expect(`echo hello`).toEvaluateTo('hello')
+  })
+})
+
+const withCommands = (commands: CommandShape[], fn: () => void) => {
+  try {
+    setCommandSource(() => commands)
+    fn()
+  } catch (e) {
+    throw e
+  } finally {
+    resetCommandSource()
+  }
+}

src/evaluator/evaluator.ts

@@ -1,36 +1,8 @@
-import { nodeToString } from '#/evaluator/treeHelper'
 import { Tree, type SyntaxNode } from '@lezer/common'
 import * as terms from '../parser/shrimp.terms.ts'
-import { errorMessage } from '#utils/utils.ts'
+import { RuntimeError } from '#evaluator/runtimeError.ts'
-
-type Context = Map<string, any>
-
-function getChildren(node: SyntaxNode): SyntaxNode[] {
-  const children = []
-  let child = node.firstChild
-  while (child) {
-    children.push(child)
-    child = child.nextSibling
-  }
-  return children
-}
-
-class RuntimeError extends Error {
-  constructor(message: string, private input: string, private from: number, private to: number) {
-    super(message)
-    this.name = 'RuntimeError'
-    this.message = `${message} at "${input.slice(from, to)}" (${from}:${to})`
-  }
-
-  toReadableString(code: string) {
-    const pointer = ' '.repeat(this.from) + '^'.repeat(this.to - this.from)
-    const context = code.split('\n').slice(-2).join('\n')
-    return `${context}\n${pointer}\n${this.message}`
-  }
-}
 
 export const evaluate = (input: string, tree: Tree, context: Context) => {
-  // Just evaluate the top-level children, don't use iterate()
   let result = undefined
   let child = tree.topNode.firstChild
   try {
@@ -42,7 +14,7 @@ export const evaluate = (input: string, tree: Tree, context: Context) => {
     if (error instanceof RuntimeError) {
       throw new Error(error.toReadableString(input))
     } else {
-      throw new RuntimeError('Unknown error during evaluation', input, 0, input.length)
+      throw new Error('Unknown error during evaluation')
     }
   }
 
@@ -50,138 +22,150 @@ export const evaluate = (input: string, tree: Tree, context: Context) => {
 }
 
 const evaluateNode = (node: SyntaxNode, input: string, context: Context): any => {
-  const value = input.slice(node.from, node.to)
+  const evalNode = syntaxNodeToEvalNode(node, input, context)
 
-  switch (node.type.id) {
+  switch (evalNode.kind) {
-    case terms.Number: {
+    case 'number':
-      return parseFloat(value)
+    case 'string':
-    }
+    case 'boolean':
+      return evalNode.value
 
-    case terms.String: {
+    case 'identifier': {
-      return value.slice(1, -1) // Remove quotes
+      const name = evalNode.name
-    }
+      if (context.has(name)) {
-    case terms.Boolean: {
+        return context.get(name)
-      return value === 'true'
+      } else {
-    }
+        throw new RuntimeError(`Undefined variable "${name}"`, node.from, node.to)
-
-    case terms.Identifier: {
-      if (!context.has(value)) {
-        throw new RuntimeError(`Undefined identifier: ${value}`, input, node.from, node.to)
-      }
-      return context.get(value)
-    }
-
-    case terms.BinOp: {
-      let [left, op, right] = getChildren(node)
-
-      left = assertNode(left, 'LeftOperand')
-      op = assertNode(op, 'Operator')
-      right = assertNode(right, 'RightOperand')
-
-      const leftValue = evaluateNode(left, input, context)
-      const opValue = input.slice(op.from, op.to)
-      const rightValue = evaluateNode(right, input, context)
-
-      switch (opValue) {
-        case '+':
-          return leftValue + rightValue
-        case '-':
-          return leftValue - rightValue
-        case '*':
-          return leftValue * rightValue
-        case '/':
-          return leftValue / rightValue
-        default:
-          throw new RuntimeError(`Unknown operator: ${opValue}`, input, op.from, op.to)
       }
     }
 
-    case terms.Assignment: {
+    case 'assignment': {
-      const [identifier, _operator, expr] = getChildren(node)
+      const name = evalNode.name
-
+      const value = evaluateNode(evalNode.value.node, input, context)
-      const identifierNode = assertNode(identifier, 'Identifier')
-      const exprNode = assertNode(expr, 'Expression')
-
-      const name = input.slice(identifierNode.from, identifierNode.to)
-      const value = evaluateNode(exprNode, input, context)
       context.set(name, value)
 
       return value
     }
 
-    case terms.Function: {
+    case 'binop': {
-      const [params, body] = getChildren(node)
+      const left = evaluateNode(evalNode.left, input, context)
+      const right = evaluateNode(evalNode.right, input, context)
 
-      const paramNodes = getChildren(assertNode(params, 'Parameters'))
+      if (evalNode.op === '+') {
-      const bodyNode = assertNode(body, 'Body')
+        return left + right
-
+      } else if (evalNode.op === '-') {
-      const paramNames = paramNodes.map((param) => {
+        return left - right
-        const paramNode = assertNode(param, 'Identifier')
+      } else if (evalNode.op === '*') {
-        return input.slice(paramNode.from, paramNode.to)
+        return left * right
-      })
+      } else if (evalNode.op === '/') {
-
+        return left / right
-      return (...args: any[]) => {
+      } else {
-        if (args.length !== paramNames.length) {
+        throw new RuntimeError(`Unsupported operator "${evalNode.op}"`, node.from, node.to)
-          throw new RuntimeError(
+      }
-            `Expected ${paramNames.length} arguments, but got ${args.length}`,
+    }
-            input,
+  }
-            node.from,
-            node.to
-          )
 }
 
-        const localContext = new Map(context)
+type Operators = '+' | '-' | '*' | '/'
-        paramNames.forEach((param, index) => {
+type Context = Map<string, any>
-          localContext.set(param, args[index])
+type EvalNode =
-        })
+  | { kind: 'number'; value: number; node: SyntaxNode }
+  | { kind: 'string'; value: string; node: SyntaxNode }
+  | { kind: 'boolean'; value: boolean; node: SyntaxNode }
+  | { kind: 'identifier'; name: string; node: SyntaxNode }
+  | { kind: 'binop'; op: Operators; left: SyntaxNode; right: SyntaxNode; node: SyntaxNode }
+  | { kind: 'assignment'; name: string; value: EvalNode; node: SyntaxNode }
+  | { kind: 'command'; name: string; args: EvalNode[]; node: SyntaxNode }
 
-        return evaluateNode(bodyNode, input, localContext)
+const syntaxNodeToEvalNode = (node: SyntaxNode, input: string, context: Context): EvalNode => {
+  const value = input.slice(node.from, node.to)
+
+  switch (node.type.id) {
+    case terms.Number:
+      return { kind: 'number', value: parseFloat(value), node }
+
+    case terms.String:
+      return { kind: 'string', value: value.slice(1, -1), node } // Remove quotes
+
+    case terms.Boolean:
+      return { kind: 'boolean', value: value === 'true', node }
+
+    case terms.Identifier:
+      return { kind: 'identifier', name: value, node }
+
+    case terms.BinOp: {
+      const [left, op, right] = destructure(node, ['*', '*', '*'])
+      const opString = input.slice(op.from, op.to) as Operators
+      return { kind: 'binop', op: opString, left, right, node }
     }
+
+    case terms.Assignment: {
+      const [identifier, _equals, expr] = destructure(node, [terms.Identifier, '*', '*'])
+
+      const name = input.slice(identifier.from, identifier.to)
+      const value = syntaxNodeToEvalNode(expr, input, context)
+
+      return { kind: 'assignment', name, value, node }
+    }
+
+    case terms.ParenExpr: {
+      const [_leftParen, expr, _rightParen] = destructure(node, ['*', '*', '*'])
+      return syntaxNodeToEvalNode(expr, input, context)
     }
 
     case terms.CommandCall: {
-      const commandNode = assertNode(node.firstChild, 'Command')
+      const [_at, identifier, _leftParen, ...rest] = destructure(node, [
-      const commandIdentifier = assertNode(commandNode.firstChild, 'Identifier')
+        '*',
-      const command = input.slice(commandIdentifier.from, commandIdentifier.to)
+        terms.Identifier,
-
+        '*',
-      const args = getChildren(node)
+        '*',
-        .slice(1)
+      ])
-        .map((argNode) => {
-          if (argNode.type.id === terms.Arg) {
-            return evaluateNode(argNode, input, context)
-          } else if (argNode.type.id === terms.NamedArg) {
-            return evaluateNode(argNode, input, context)
-          } else {
-            throw new RuntimeError(
-              `Unexpected argument type: ${argNode.type.name}`,
-              input,
-              argNode.from,
-              argNode.to
-            )
-          }
-        })
-      const commandName = input.slice(commandIdentifier.from, commandIdentifier.to)
   }
 
-    default:
+  throw new RuntimeError(`Unsupported node type "${node.type.name}"`, node.from, node.to)
-      const isLowerCase = node.type.name[0] == node.type.name[0]?.toLowerCase()
+}
 
-      // Ignore nodes with lowercase names, those are for syntax only
+/* 
-      if (!isLowerCase) {
+The code below is a...
+SIN AGAINST GOD!
+...but it makes it easier to use above
+*/
+type ExpectedType = '*' | number
+function destructure(node: SyntaxNode, expected: [ExpectedType]): [SyntaxNode]
+function destructure(
+  node: SyntaxNode,
+  expected: [ExpectedType, ExpectedType]
+): [SyntaxNode, SyntaxNode]
+function destructure(
+  node: SyntaxNode,
+  expected: [ExpectedType, ExpectedType, ExpectedType]
+): [SyntaxNode, SyntaxNode, SyntaxNode]
+function destructure(node: SyntaxNode, expected: ExpectedType[]): SyntaxNode[] {
+  const children: SyntaxNode[] = []
+  let child = node.firstChild
+  while (child) {
+    children.push(child)
+    child = child.nextSibling
+  }
+
+  if (children.length !== expected.length) {
     throw new RuntimeError(
-          `Unsupported node type "${node.type.name}"`,
+      `${node.type.name} expected ${expected.length} children, got ${children.length}`,
-          input,
       node.from,
       node.to
     )
   }
-  }
-}
 
-const assertNode = (node: any, expectedName: string): SyntaxNode => {
+  children.forEach((child, i) => {
-  if (!node) {
+    const expectedType = expected[i]
-    throw new Error(`Expected "${expectedName}", but got undefined`)
+    if (expectedType !== '*' && child.type.id !== expectedType) {
+      throw new RuntimeError(
+        `Child ${i} of ${node.type.name} expected ${expectedType}, got ${child.type.id} (${child.type.name})`,
+        child.from,
+        child.to
+      )
     }
+  })
 
-  return node
+  return children
 }

src/evaluator/runtimeError.ts

@@ -0,0 +1,16 @@
+export class RuntimeError extends Error {
+  constructor(message: string, private from: number, private to: number) {
+    super(message)
+    this.name = 'RuntimeError'
+    this.message = message
+  }
+
+  toReadableString(input: string) {
+    const pointer = ' '.repeat(this.from) + '^'.repeat(this.to - this.from)
+    const message = `${this.message} at "${input.slice(this.from, this.to)}" (${this.from}:${
+      this.to
+    })`
+
+    return `${input}\n${pointer}\n${message}`
+  }
+}

src/parser/highlight.js

@@ -1,5 +1,4 @@
 import { styleTags, tags } from '@lezer/highlight'
-import { Command, Identifier, Params } from '#/parser/shrimp.terms'
 
 export const highlighting = styleTags({
   Identifier: tags.name,

src/parser/shrimp.grammar

@@ -75,4 +75,5 @@ BinOp {
   expr !additive "-" expr
 }
 
-atom { Identifier ~command | Number | String | Boolean | leftParen expr rightParen }
+ParenExpr { leftParen expr rightParen }
+atom { Identifier ~command | Number | String | Boolean | ParenExpr }

src/parser/shrimp.terms.ts

@@ -4,7 +4,7 @@ export const
   Command = 2,
   CommandPartial = 3,
   UnquotedArg = 4,
-  insertedSemi = 30,
+  insertedSemi = 31,
   Program = 5,
   CommandCall = 6,
   NamedArg = 7,
@@ -12,13 +12,14 @@ export const
   Number = 9,
   String = 10,
   Boolean = 11,
-  leftParen = 12,
+  ParenExpr = 12,
-  Assignment = 13,
+  leftParen = 13,
-  equals = 14,
+  Assignment = 14,
-  Function = 15,
+  equals = 15,
-  fn = 16,
+  Function = 16,
-  Params = 17,
+  fn = 17,
-  BinOp = 19,
+  Params = 18,
-  rightParen = 24,
+  BinOp = 20,
-  PartialNamedArg = 25,
+  rightParen = 25,
-  Arg = 26
+  PartialNamedArg = 26,
+  Arg = 27

src/parser/shrimp.test.ts

@@ -1,331 +1,331 @@
-import { expect, describe, test } from 'bun:test'
+// import { expect, describe, test } from 'bun:test'
-import { afterEach } from 'bun:test'
+// import { afterEach } from 'bun:test'
-import { resetCommandSource, setCommandSource } from '#editor/commands'
+// import { resetCommandSource, setCommandSource } from '#editor/commands'
-import { beforeEach } from 'bun:test'
+// import { beforeEach } from 'bun:test'
-import './shrimp.grammar' // Importing this so changes cause it to retest!
+// import './shrimp.grammar' // Importing this so changes cause it to retest!
 
-describe('calling commands', () => {
+// describe('calling commands', () => {
-  beforeEach(() => {
+//   beforeEach(() => {
-    setCommandSource(() => [
+//     setCommandSource(() => [
-      { command: 'tail', args: [{ name: 'path', type: 'string' }] },
+//       { command: 'tail', args: [{ name: 'path', type: 'string' }] },
-      { command: 'head', args: [{ name: 'path', type: 'string' }] },
+//       { command: 'head', args: [{ name: 'path', type: 'string' }] },
-      { command: 'echo', args: [{ name: 'path', type: 'string' }] },
+//       { command: 'echo', args: [{ name: 'path', type: 'string' }] },
-    ])
+//     ])
-  })
+//   })
 
-  afterEach(() => {
+//   afterEach(() => {
-    resetCommandSource()
+//     resetCommandSource()
-  })
+//   })
 
-  test('basic', () => {
+//   test('basic', () => {
-    expect('tail path').toMatchTree(`
+//     expect('tail path').toMatchTree(`
-      CommandCall
+//       CommandCall
-        Command tail
+//         Command tail
-        Arg
+//         Arg
-          Identifier path
+//           Identifier path
-    `)
+//     `)
 
-    expect('tai').toMatchTree(`
+//     expect('tai').toMatchTree(`
-      CommandCall
+//       CommandCall
-        CommandPartial tai
+//         CommandPartial tai
-    `)
+//     `)
-  })
+//   })
 
-  test('command with arg that is also a command', () => {
+//   test('command with arg that is also a command', () => {
-    expect('tail tail').toMatchTree(`
+//     expect('tail tail').toMatchTree(`
-      CommandCall
+//       CommandCall
-        Command tail
+//         Command tail
-        Arg
+//         Arg
-          Identifier tail
+//           Identifier tail
-    `)
+//     `)
 
-    expect('tai').toMatchTree(`
+//     expect('tai').toMatchTree(`
-      CommandCall
+//       CommandCall
-        CommandPartial tai
+//         CommandPartial tai
-    `)
+//     `)
-  })
+//   })
 
-  test('when no commands match, falls back to Identifier', () => {
+//   test('when no commands match, falls back to Identifier', () => {
-    expect('omgwtf').toMatchTree(`
+//     expect('omgwtf').toMatchTree(`
-      Identifier omgwtf
+//       Identifier omgwtf
-    `)
+//     `)
-  })
+//   })
 
-  // In shrimp.test.ts, add to the 'calling commands' section
+//   // In shrimp.test.ts, add to the 'calling commands' section
-  test('arg', () => {
+//   test('arg', () => {
-    expect('tail l').toMatchTree(`    
+//     expect('tail l').toMatchTree(`
-      CommandCall
+//       CommandCall
-        Command tail
+//         Command tail
-        Arg
+//         Arg
-          Identifier l
+//           Identifier l
-  `)
+//   `)
-  })
+//   })
 
-  test('partial namedArg', () => {
+//   test('partial namedArg', () => {
-    expect('tail lines=').toMatchTree(`    
+//     expect('tail lines=').toMatchTree(`
-      CommandCall
+//       CommandCall
-        Command tail
+//         Command tail
-        PartialNamedArg
+//         PartialNamedArg
-          NamedArgPrefix lines=
+//           NamedArgPrefix lines=
-    `)
+//     `)
-  })
+//   })
 
-  test('complete namedArg', () => {
+//   test('complete namedArg', () => {
-    expect('tail lines=10').toMatchTree(`    
+//     expect('tail lines=10').toMatchTree(`
-      CommandCall
+//       CommandCall
-        Command tail
+//         Command tail
-        NamedArg
+//         NamedArg
-          NamedArgPrefix lines=
+//           NamedArgPrefix lines=
-          Number 10
+//           Number 10
-    `)
+//     `)
-  })
+//   })
 
-  test('mixed positional and named args', () => {
+//   test('mixed positional and named args', () => {
-    expect('tail ../file.txt lines=5').toMatchTree(`    
+//     expect('tail ../file.txt lines=5').toMatchTree(`
-      CommandCall
+//       CommandCall
-        Command tail
+//         Command tail
-        Arg
+//         Arg
-          UnquotedArg ../file.txt
+//           UnquotedArg ../file.txt
-        NamedArg
+//         NamedArg
-          NamedArgPrefix lines=
+//           NamedArgPrefix lines=
-          Number 5
+//           Number 5
-  `)
+//   `)
-  })
+//   })
 
-  test('named args', () => {
+//   test('named args', () => {
-    expect(`tail lines='5' path`).toMatchTree(`
+//     expect(`tail lines='5' path`).toMatchTree(`
-      CommandCall
+//       CommandCall
-        Command tail
+//         Command tail
-        NamedArg
+//         NamedArg
-          NamedArgPrefix lines=
+//           NamedArgPrefix lines=
-          String 5
+//           String 5
-        Arg
+//         Arg
-          Identifier path
+//           Identifier path
-    `)
+//     `)
-  })
+//   })
 
-  test('complex args', () => {
+//   test('complex args', () => {
-    expect(`tail lines=(2 + 3) filter='error' (a + b)`).toMatchTree(`
+//     expect(`tail lines=(2 + 3) filter='error' (a + b)`).toMatchTree(`
-      CommandCall
+//       CommandCall
-        Command tail
+//         Command tail
-        NamedArg
+//         NamedArg
-          NamedArgPrefix lines=
+//           NamedArgPrefix lines=
-          paren (
+//           paren (
-          BinOp
+//           BinOp
-            Number 2
+//             Number 2
-            operator +
+//             operator +
-            Number 3
+//             Number 3
-          paren )
+//           paren )
-        NamedArg
+//         NamedArg
-          NamedArgPrefix filter=
+//           NamedArgPrefix filter=
-          String error
+//           String error
 
-        Arg
+//         Arg
-          paren (
+//           paren (
-          BinOp
+//           BinOp
-            Identifier a
+//             Identifier a
-            operator +
+//             operator +
-            Identifier b
+//             Identifier b
-          paren )
+//           paren )
-    `)
+//     `)
-  })
+//   })
-})
+// })
 
-describe('Identifier', () => {
+// describe('Identifier', () => {
-  test('parses simple identifiers', () => {
+//   test('parses simple identifiers', () => {
-    expect('hyphenated-var').toMatchTree(`Identifier hyphenated-var`)
+//     expect('hyphenated-var').toMatchTree(`Identifier hyphenated-var`)
-    expect('var').toMatchTree(`Identifier var`)
+//     expect('var').toMatchTree(`Identifier var`)
-    expect('var123').toMatchTree(`Identifier var123`)
+//     expect('var123').toMatchTree(`Identifier var123`)
-  })
+//   })
 
-  test('fails on underscores and capital letters', () => {
+//   test('fails on underscores and capital letters', () => {
-    expect('myVar').toFailParse()
+//     expect('myVar').toFailParse()
-    expect('underscore_var').toFailParse()
+//     expect('underscore_var').toFailParse()
-    expect('_leadingUnderscore').toFailParse()
+//     expect('_leadingUnderscore').toFailParse()
-    expect('trailingUnderscore_').toFailParse()
+//     expect('trailingUnderscore_').toFailParse()
-    expect('mixed-123_var').toFailParse()
+//     expect('mixed-123_var').toFailParse()
-  })
+//   })
 
-  test('parses identifiers with emojis', () => {
+//   test('parses identifiers with emojis', () => {
-    expect('var😊').toMatchTree(`Identifier var😊`)
+//     expect('var😊').toMatchTree(`Identifier var😊`)
-    expect('😊').toMatchTree(`Identifier 😊`)
+//     expect('😊').toMatchTree(`Identifier 😊`)
-  })
+//   })
-})
+// })
 
-describe('BinOp', () => {
+// describe('BinOp', () => {
-  test('addition tests', () => {
+//   test('addition tests', () => {
-    expect('2 + 3').toMatchTree(`
+//     expect('2 + 3').toMatchTree(`
-      BinOp
+//       BinOp
-        Number 2
+//         Number 2
-        operator +
+//         operator +
-        Number 3
+//         Number 3
-    `)
+//     `)
-  })
+//   })
 
-  test('subtraction tests', () => {
+//   test('subtraction tests', () => {
-    expect('5 - 2').toMatchTree(`
+//     expect('5 - 2').toMatchTree(`
-      BinOp
+//       BinOp
-        Number 5
+//         Number 5
-        operator -
+//         operator -
-        Number 2
+//         Number 2
-    `)
+//     `)
-  })
+//   })
 
-  test('multiplication tests', () => {
+//   test('multiplication tests', () => {
-    expect('4 * 3').toMatchTree(`
+//     expect('4 * 3').toMatchTree(`
-      BinOp
+//       BinOp
-        Number 4
+//         Number 4
-        operator *
+//         operator *
-        Number 3
+//         Number 3
-    `)
+//     `)
-  })
+//   })
 
-  test('division tests', () => {
+//   test('division tests', () => {
-    expect('8 / 2').toMatchTree(`
+//     expect('8 / 2').toMatchTree(`
-      BinOp
+//       BinOp
-        Number 8
+//         Number 8
-        operator /
+//         operator /
-        Number 2
+//         Number 2
-    `)
+//     `)
-  })
+//   })
 
-  test('mixed operations with precedence', () => {
+//   test('mixed operations with precedence', () => {
-    expect('2 + 3 * 4 - 5 / 1').toMatchTree(`
+//     expect('2 + 3 * 4 - 5 / 1').toMatchTree(`
-      BinOp
+//       BinOp
-        BinOp
+//         BinOp
-          Number 2
+//           Number 2
-          operator +
+//           operator +
-          BinOp
+//           BinOp
-            Number 3
+//             Number 3
-            operator *
+//             operator *
-            Number 4
+//             Number 4
-        operator -
+//         operator -
-        BinOp
+//         BinOp
-          Number 5
+//           Number 5
-          operator /
+//           operator /
-          Number 1
+//           Number 1
-    `)
+//     `)
-  })
+//   })
-})
+// })
 
-describe('Fn', () => {
+// describe('Fn', () => {
-  test('parses function with single parameter', () => {
+//   test('parses function with single parameter', () => {
-    expect('fn x: x + 1').toMatchTree(`
+//     expect('fn x: x + 1').toMatchTree(`
-      Function
+//       Function
-        keyword fn
+//         keyword fn
-        Params
+//         Params
-          Identifier x
+//           Identifier x
-        colon :
+//         colon :
-        BinOp
+//         BinOp
-          Identifier x
+//           Identifier x
-          operator +
+//           operator +
-          Number 1`)
+//           Number 1`)
-  })
+//   })
 
-  test('parses function with multiple parameters', () => {
+//   test('parses function with multiple parameters', () => {
-    expect('fn x y: x * y').toMatchTree(`
+//     expect('fn x y: x * y').toMatchTree(`
-      Function
+//       Function
-        keyword fn
+//         keyword fn
-        Params
+//         Params
-          Identifier x
+//           Identifier x
-          Identifier y
+//           Identifier y
-        colon :
+//         colon :
-        BinOp
+//         BinOp
-          Identifier x
+//           Identifier x
-          operator *
+//           operator *
-          Identifier y`)
+//           Identifier y`)
-  })
+//   })
 
-  test('parses nested functions', () => {
+//   test('parses nested functions', () => {
-    expect('fn x: fn y: x + y').toMatchTree(`
+//     expect('fn x: fn y: x + y').toMatchTree(`
-      Function
+//       Function
-        keyword fn
+//         keyword fn
-        Params
+//         Params
-          Identifier x
+//           Identifier x
-        colon :
+//         colon :
-        Function
+//         Function
-          keyword fn
+//           keyword fn
-          Params
+//           Params
-            Identifier y
+//             Identifier y
-          colon :
+//           colon :
-          BinOp
+//           BinOp
-            Identifier x
+//             Identifier x
-            operator +
+//             operator +
-            Identifier y`)
+//             Identifier y`)
-  })
+//   })
-})
+// })
 
-describe('Identifier', () => {
+// describe('Identifier', () => {
-  test('parses hyphenated identifiers correctly', () => {
+//   test('parses hyphenated identifiers correctly', () => {
-    expect('my-var').toMatchTree(`Identifier my-var`)
+//     expect('my-var').toMatchTree(`Identifier my-var`)
-    expect('double--trouble').toMatchTree(`Identifier double--trouble`)
+//     expect('double--trouble').toMatchTree(`Identifier double--trouble`)
-  })
+//   })
-})
+// })
 
-describe('Assignment', () => {
+// describe('Assignment', () => {
-  test('parses assignment with addition', () => {
+//   test('parses assignment with addition', () => {
-    expect('x = 5 + 3').toMatchTree(`
+//     expect('x = 5 + 3').toMatchTree(`
-      Assignment
+//       Assignment
-        Identifier x
+//         Identifier x
-        operator =
+//         operator =
-        BinOp
+//         BinOp
-          Number 5
+//           Number 5
-          operator +
+//           operator +
-          Number 3`)
+//           Number 3`)
-  })
+//   })
 
-  test('parses assignment with functions', () => {
+//   test('parses assignment with functions', () => {
-    expect('add = fn a b: a + b').toMatchTree(`
+//     expect('add = fn a b: a + b').toMatchTree(`
-      Assignment
+//       Assignment
-        Identifier add
+//         Identifier add
-        operator =
+//         operator =
-        Function
+//         Function
-          keyword fn
+//           keyword fn
-          Params
+//           Params
-            Identifier a
+//             Identifier a
-            Identifier b
+//             Identifier b
-          colon :
+//           colon :
-          BinOp
+//           BinOp
-            Identifier a
+//             Identifier a
-            operator +
+//             operator +
-            Identifier b`)
+//             Identifier b`)
-  })
+//   })
-})
+// })
 
-describe('Parentheses', () => {
+// describe('Parentheses', () => {
-  test('parses expressions with parentheses correctly', () => {
+//   test('parses expressions with parentheses correctly', () => {
-    expect('(2 + 3) * 4').toMatchTree(`
+//     expect('(2 + 3) * 4').toMatchTree(`
-      BinOp
+//       BinOp
-        paren (
+//         paren (
-        BinOp
+//         BinOp
-          Number 2
+//           Number 2
-          operator +
+//           operator +
-          Number 3
+//           Number 3
-        paren )
+//         paren )
-        operator *
+//         operator *
-        Number 4`)
+//         Number 4`)
-  })
+//   })
 
-  test('parses nested parentheses correctly', () => {
+//   test('parses nested parentheses correctly', () => {
-    expect('((1 + 2) * (3 - 4)) / 5').toMatchTree(`
+//     expect('((1 + 2) * (3 - 4)) / 5').toMatchTree(`
-      BinOp
+//       BinOp
-        paren (
+//         paren (
-        BinOp
+//         BinOp
-          paren (
+//           paren (
-          BinOp
+//           BinOp
-            Number 1
+//             Number 1
-            operator +
+//             operator +
-            Number 2
+//             Number 2
-          paren )
+//           paren )
-          operator *
+//           operator *
-          paren (
+//           paren (
-          BinOp
+//           BinOp
-            Number 3
+//             Number 3
-            operator -
+//             operator -
-            Number 4
+//             Number 4
-          paren )
+//           paren )
-        paren )
+//         paren )
-        operator /
+//         operator /
-        Number 5`)
+//         Number 5`)
-  })
+//   })
-})
+// })

src/parser/shrimp.ts

@@ -4,15 +4,15 @@ import {tokenizer, argTokenizer, insertSemicolon} from "./tokenizers"
 import {highlighting} from "./highlight.js"
 export const parser = LRParser.deserialize({
   version: 14,
-  states: "%^OkQTOOOuQaO'#DPO!aQTO'#CkO!iQaO'#C}OOQ`'#DQ'#DQOOQl'#DP'#DPOVQTO'#DPO#fQnO'#CbO!uQaO'#C}QOQPOOOVQTO,59TOOQS'#Cx'#CxO#pQTO'#CmO#xQPO,59VOVQTO,59ZOVQTO,59ZOOQO'#DR'#DROOQO,59i,59iO#}QPO,59kOOQl'#DO'#DOO$tQnO'#CuOOQl'#Cv'#CvOOQl'#Cw'#CwO%RQnO,58|O%]QaO1G.oOOQS-E6v-E6vOVQTO1G.qOOQ`1G.u1G.uO%tQaO1G.uOOQl1G/V1G/VOOQl,58},58}OOQl-E6u-E6uO&]QaO7+$]",
+  states: "%^OkQTOOOuQaO'#DQO!aQTO'#ClO!iQaO'#DOOOQ`'#DR'#DROVQTO'#ChOOQl'#DQ'#DQO#fQnO'#CbO!uQaO'#DOQOQPOOOVQTO,59UOOQS'#Cy'#CyO#pQTO'#CnO#xQPO,59WOVQTO,59[OVQTO,59[OOQO'#DS'#DSOOQO,59j,59jO#}QPO,59SOOQl'#DP'#DPO$tQnO'#CvOOQl'#Cw'#CwOOQl'#Cx'#CxO%RQnO,58|O%]QaO1G.pOOQS-E6w-E6wOVQTO1G.rOOQ`1G.v1G.vO%tQaO1G.vOOQl1G.n1G.nOOQl,58},58}OOQl-E6v-E6vO&]QaO7+$^",
-  stateData: "&w~OpOS~OPPOXTOYTOZTO[UO`QO~OQVORVO~PVO^YOdsXesXfsXgsXnsXvsXhsX~OPZObaP~Od^Oe^Of_Og_On`Ov`O~OPTOScOWdOXTOYTOZTO[UO~OnUXvUX~P!}OPZObaX~ObjO~Od^Oe^Of_Og_OhmO~OPTOScOXTOYTOZTO[UO~OWiXniXviX~P$`OnUavUa~P!}Od^Oe^Of_Og_On]iv]ih]i~Od^Oe^Ofcigcincivcihci~Od^Oe^Of_Og_On_qv_qh_q~OXg~",
+  stateData: "&w~OqOS~OPPOXUOYUOZUO]TOaQO~OQVORVO~PVO_YOetXftXgtXhtXotXwtXitX~OPZOcbP~Oe^Of^Og_Oh_Oo`Ow`O~OPUOScOWdOXUOYUOZUO]TO~OoUXwUX~P!}OPZOcbX~OcjO~Oe^Of^Og_Oh_OimO~OPUOScOXUOYUOZUO]TO~OWjXojXwjX~P$`OoUawUa~P!}Oe^Of^Og_Oh_Oo^iw^ii^i~Oe^Of^Ogdihdiodiwdiidi~Oe^Of^Og_Oh_Oo`qw`qi`q~OXh~",
-  goto: "#fvPPPPPPwzPPPPP!OP!OP!WP!OPPPPPzz!Z!aPPPP!g!j!q#O#bRWOTfVg]SOUY^_jR]QQgVRogQ[QRi[RXOSeVgRnd[SOUY^_jVcVdgQROQbUQhYQk^Ql_RpjTaRW",
+  goto: "#rwPPPPPPx{PPPP!PP![P![P!dP![PPPPP{{!g!mPPPP!s!v!}#[#nRWOTfVgcUOTVY^_dgj]SOTY^_jR]QQgVRogQ[QRi[RXOSeVgRnd[SOTY^_jVcVdgQROQbTQhYQk^Ql_RpjTaRW",
-  nodeNames: "⚠ Identifier Command CommandPartial UnquotedArg Program CommandCall NamedArg NamedArgPrefix Number String Boolean paren Assignment operator Function keyword Params colon BinOp operator operator operator operator paren PartialNamedArg Arg",
+  nodeNames: "⚠ Identifier Command CommandPartial UnquotedArg Program CommandCall NamedArg NamedArgPrefix Number String Boolean ParenExpr paren Assignment operator Function keyword Params colon BinOp operator operator operator operator paren PartialNamedArg Arg",
-  maxTerm: 38,
+  maxTerm: 39,
   propSources: [highlighting],
   skippedNodes: [0],
   repeatNodeCount: 2,
-  tokenData: "*W~RjX^!spq!swx#hxy$lyz$qz{$v{|${}!O%Q!P!Q%s!Q![%Y![!]%x!]!^%}!_!`&S#T#Y&X#Y#Z&m#Z#h&X#h#i)[#i#o&X#y#z!s$f$g!s#BY#BZ!s$IS$I_!s$I|$JO!s$JT$JU!s$KV$KW!s&FU&FV!s~!xYp~X^!spq!s#y#z!s$f$g!s#BY#BZ!s$IS$I_!s$I|$JO!s$JT$JU!s$KV$KW!s&FU&FV!s~#kUOr#hsw#hwx#}x;'S#h;'S;=`$f<%lO#h~$SUY~Or#hsw#hwx#}x;'S#h;'S;=`$f<%lO#h~$iP;=`<%l#h~$qO[~~$vOh~~${Od~~%QOf~~%VPg~!Q![%Y~%_QX~!O!P%e!Q![%Y~%hP!Q![%k~%pPX~!Q![%k~%xOe~~%}Ob~~&SOv~~&XO^~Q&[S}!O&X!Q![&X!_!`&h#T#o&XQ&mOWQ~&pV}!O&X!Q![&X!_!`&h#T#U'V#U#b&X#b#c(y#c#o&X~'YU}!O&X!Q![&X!_!`&h#T#`&X#`#a'l#a#o&X~'oU}!O&X!Q![&X!_!`&h#T#g&X#g#h(R#h#o&X~(UU}!O&X!Q![&X!_!`&h#T#X&X#X#Y(h#Y#o&X~(mSZ~}!O&X!Q![&X!_!`&h#T#o&XR)OS`P}!O&X!Q![&X!_!`&h#T#o&X~)_U}!O&X!Q![&X!_!`&h#T#f&X#f#g)q#g#o&X~)tU}!O&X!Q![&X!_!`&h#T#i&X#i#j(R#j#o&X",
+  tokenData: "*W~RjX^!spq!swx#hxy$lyz$qz{$v{|${}!O%Q!P!Q%s!Q![%Y![!]%x!]!^%}!_!`&S#T#Y&X#Y#Z&m#Z#h&X#h#i)[#i#o&X#y#z!s$f$g!s#BY#BZ!s$IS$I_!s$I|$JO!s$JT$JU!s$KV$KW!s&FU&FV!s~!xYq~X^!spq!s#y#z!s$f$g!s#BY#BZ!s$IS$I_!s$I|$JO!s$JT$JU!s$KV$KW!s&FU&FV!s~#kUOr#hsw#hwx#}x;'S#h;'S;=`$f<%lO#h~$SUY~Or#hsw#hwx#}x;'S#h;'S;=`$f<%lO#h~$iP;=`<%l#h~$qO]~~$vOi~~${Oe~~%QOg~~%VPh~!Q![%Y~%_QX~!O!P%e!Q![%Y~%hP!Q![%k~%pPX~!Q![%k~%xOf~~%}Oc~~&SOw~~&XO_~Q&[S}!O&X!Q![&X!_!`&h#T#o&XQ&mOWQ~&pV}!O&X!Q![&X!_!`&h#T#U'V#U#b&X#b#c(y#c#o&X~'YU}!O&X!Q![&X!_!`&h#T#`&X#`#a'l#a#o&X~'oU}!O&X!Q![&X!_!`&h#T#g&X#g#h(R#h#o&X~(UU}!O&X!Q![&X!_!`&h#T#X&X#X#Y(h#Y#o&X~(mSZ~}!O&X!Q![&X!_!`&h#T#o&XR)OSaP}!O&X!Q![&X!_!`&h#T#o&X~)_U}!O&X!Q![&X!_!`&h#T#f&X#f#g)q#g#o&X~)tU}!O&X!Q![&X!_!`&h#T#i&X#i#j(R#j#o&X",
   tokenizers: [0, 1, tokenizer, argTokenizer, insertSemicolon],
   topRules: {"Program":[0,5]},
   tokenPrec: 266

src/testSetup.ts

@@ -128,7 +128,7 @@ expect.extend({
       }
     } catch (error) {
       return {
-        message: () => `Evaluation threw an error: ${(error as Error).message}`,
+        message: () => `Evaluation threw an error:\n${(error as Error).message}`,
         pass: false,
       }
     }