# Shrimp Parser - Development Context

## Overview

Building a command-line language parser using Lezer (CodeMirror's parser system) with TypeScript. The goal is to create a prototype that can parse commands with arguments, similar to shell syntax, with inline hints for autocompletion.

## Current Architecture

### Grammar Structure (`shrimp.grammar`)

- **Commands**: Can be complete (`Command`) or partial (`CommandPartial`) for autocomplete
- **Arguments**: Positional or named (with `name=value` syntax)
- **Key Challenge**: Handling arbitrary text (like file paths) as arguments without conflicting with operators/keywords

### Tokenizer Setup (`tokenizers.ts`)

- **Main tokenizer**: Returns `Command`, `CommandPartial`, or `Identifier` based on context
- **Command matching**: Uses `matchCommand()` to check against available commands
- **Context-aware**: Uses `stack.canShift()` to return appropriate token based on parse position
- **Issue**: Second occurrence of command name (e.g., `tail tail`) should be `Identifier` not `Command`

### Key Design Decisions

1. **External tokenizers over regular tokens** for commands to enable:

   - Dynamic command list (can change at runtime)
   - Partial matching for autocomplete
   - Context-aware tokenization

2. **Virtual semicolons** for statement boundaries:

   - Using `insertSemicolon` external tokenizer
   - Inserts at newlines/EOF to keep parser "inside" CommandCall
   - Prevents `tail t` from parsing as two separate commands

3. **UnquotedArg token** for paths/arbitrary text:
   - Accepts anything except whitespace/parens/equals
   - Only valid in command argument context
   - Avoids conflicts with operators elsewhere

### Current Problems

1. **Parser completes CommandCall too early**

   - After `tail `, cursor shows position in `Program` not `CommandCall`
   - Makes hint system harder to implement

2. **Command token in wrong context**

   - `tail tail` - second "tail" returns `Command` token but should be `Identifier`
   - Need better context checking in tokenizer

3. **Inline hints need to be smarter**
   - Must look backward to find command context
   - Handle cases where parser has "completed" the command

### Test Infrastructure

- Custom test matchers: `toMatchTree`, `toEvaluateTo`
- Command source injection for testing: `setCommandSource()`
- Tests in `shrimp.test.ts`

### File Structure

```
src/parser/
  shrimp.grammar     - Lezer grammar definition
  tokenizers.ts      - External tokenizers
  shrimp.ts         - Generated parser

src/editor/
  commands.ts        - Command definitions
  plugins/
    inlineHints.tsx  - Autocomplete hint UI
```

## Next Steps

1. Fix tokenizer context checking with `stack.canShift()`
2. Improve hint detection for "after command with space" case
3. Consider if grammar structure changes would help

## Key Concepts to Remember

- Lezer is LR parser - builds tree bottom-up
- External tokenizers run at each position
- `@skip { space }` makes whitespace invisible to parser
- Token precedence matters for overlap resolution
- `stack.canShift(tokenId)` checks if token is valid at current position