shrimp/vscode-extension/docs/plans/2025-01-24-autocomplete-design.md

Autocomplete Design for Shrimp VSCode Extension

Date: 2025-01-24 Status: Approved, ready for implementation

Goal

Add intelligent autocomplete for Shrimp prelude functions and dollar properties in the VSCode extension.

Scope

Phase 1 (This Implementation)

• Module completions: dict.*, list.*, str.* member functions
• Dollar completions: $.* runtime properties
• Trigger behavior:
  • Auto-trigger on . for module/dollar members
  • Manual trigger (Ctrl+Space) for keywords (existing behavior)

Future Phases (Deferred)

• Top-level prelude globals (echo, type, inspect, etc.)
• Completion descriptions and documentation
• Chained property access (e.g., dict.keys.)

Requirements

1. Show function signatures with parameter names (e.g., get(dict, key))
2. Use proper VSCode completion kinds (Method for functions, Property for dollar)
3. Parser-based context detection (no regex hacks)
4. TypeScript metadata for easy maintenance and editor support
5. Minimal, readable code

Architecture

File Structure

vscode-extension/
├── server/src/
│   ├── metadata/
│   │   └── prelude-completions.ts    # TypeScript metadata
│   ├── completion/
│   │   ├── completionProvider.ts     # Main completion logic
│   │   └── contextAnalyzer.ts        # Parser-based context detection
│   └── server.ts                      # Wire up completion handler

Data Flow

1. User types dict. → VSCode sends onCompletion request with cursor position
2. contextAnalyzer uses Shrimp parser to determine context:
   • Is cursor in a DotGet node?
   • What's on the left side of the dot?
3. completionProvider returns appropriate completions based on context:
   • Module functions if dict., list., or str.
   • Dollar properties if $.
   • Empty list otherwise (fall back to keywords)

Parser Behavior Analysis

Test Results

We tested how incomplete and partial DotGet expressions parse:

Input	Parse Result	Node at cursor
dict.	DotGet(IdentifierBeforeDot, ⚠)	DotGet
dict.g	DotGet(IdentifierBeforeDot, Identifier)	Identifier
dict.get	DotGet(IdentifierBeforeDot, Identifier)	Identifier
$.	DotGet(Dollar, ⚠)	DotGet
$.e	DotGet(Dollar, Identifier)	Identifier
$.env	DotGet(Dollar, Identifier)	Identifier

Key Insights:

• Incomplete DotGet (dict., $.) → resolveInner returns DotGet node directly
• Partial identifier (dict.g, $.e) → resolveInner returns Identifier node (parent is DotGet)
• Error node (⚠) marks incomplete input, perfect for triggering autocomplete

Context Detection Strategy

Handle both cases:

1. Node type is DotGet → cursor right after dot
2. Node type is Identifier with DotGet parent → typing property name

Extract left side of dot to determine if it's a known module or $.

Implementation Details

1. Metadata Structure

File: server/src/metadata/prelude-completions.ts

export type CompletionMetadata = {
  params: string[]
  description?: string  // Optional for now, add later
}

export const completions = {
  modules: {
    dict: {
      'get': { params: ['dict', 'key'] },
      'set': { params: ['dict', 'key', 'value'] },
      'keys': { params: ['dict'] },
      'values': { params: ['dict'] },
      'has?': { params: ['dict', 'key'] },
    },
    list: {
      'map': { params: ['list', 'fn'] },
      'filter': { params: ['list', 'fn'] },
      'reduce': { params: ['list', 'fn', 'initial'] },
      'reverse': { params: ['list'] },
      'length': { params: ['list'] },
    },
    str: {
      'split': { params: ['str', 'separator'] },
      'join': { params: ['list', 'separator'] },
      'trim': { params: ['str'] },
      'uppercase': { params: ['str'] },
      'lowercase': { params: ['str'] },
    },
  },
  dollar: {
    'env': { params: [] },
    'pid': { params: [] },
    'args': { params: [] },
    'argv': { params: [] },
    'cwd': { params: [] },
    'script': { params: [] },
  },
} as const

Design decisions:

• TypeScript (not JSON) for editor autocomplete while editing metadata
• as const for type safety
• Single export with nested structure (completions.modules, completions.dollar)
• Start with ~20 most common functions per module

2. Context Analyzer

File: server/src/completion/contextAnalyzer.ts

import { TextDocument } from 'vscode-languageserver-textdocument'
import { SyntaxNode } from '@lezer/common'
import { parser } from '../../../src/parser/shrimp'
import * as Terms from '../../../src/parser/shrimp.terms'

export type CompletionContext =
  | { type: 'module', moduleName: string }
  | { type: 'dollar' }
  | { type: 'none' }

export const analyzeCompletionContext = (
  document: TextDocument,
  position: { line: number, character: number }
): CompletionContext => {
  const offset = document.offsetAt(position)
  const text = document.getText()
  const tree = parser.parse(text)

  // Find node at cursor - could be DotGet or Identifier inside DotGet
  let node = tree.resolveInner(offset, -1)

  // Case 1: Incomplete DotGet (dict. or $.)
  // resolveInner returns DotGet node directly
  if (node.type.id === Terms.DotGet) {
    const leftSide = extractLeftSide(node, text)
    if (leftSide === '$') return { type: 'dollar' }
    if (['dict', 'list', 'str'].includes(leftSide)) {
      return { type: 'module', moduleName: leftSide }
    }
  }

  // Case 2: Partial identifier (dict.g or $.e)
  // resolveInner returns Identifier, parent is DotGet
  if (node.type.id === Terms.Identifier && node.parent?.type.id === Terms.DotGet) {
    const dotGetNode = node.parent
    const leftSide = extractLeftSide(dotGetNode, text)
    if (leftSide === '$') return { type: 'dollar' }
    if (['dict', 'list', 'str'].includes(leftSide)) {
      return { type: 'module', moduleName: leftSide }
    }
  }

  return { type: 'none' }
}

const extractLeftSide = (dotGetNode: SyntaxNode, text: string): string => {
  const firstChild = dotGetNode.firstChild
  if (!firstChild) return ''
  return text.slice(firstChild.from, firstChild.to)
}

Design decisions:

• Use Shrimp parser (not regex) for accurate token detection
• Return typed context for type-safe consumption
• Handle both incomplete and partial completion cases
• Simple left-side extraction from first child of DotGet

3. Completion Provider

File: server/src/completion/completionProvider.ts

import { CompletionItem, CompletionItemKind } from 'vscode-languageserver/node'
import { TextDocument } from 'vscode-languageserver-textdocument'
import { completions } from '../metadata/prelude-completions'
import { analyzeCompletionContext } from './contextAnalyzer'

export const provideCompletions = (
  document: TextDocument,
  position: { line: number, character: number }
): CompletionItem[] => {
  const context = analyzeCompletionContext(document, position)

  if (context.type === 'module') {
    return buildModuleCompletions(context.moduleName)
  }

  if (context.type === 'dollar') {
    return buildDollarCompletions()
  }

  return [] // No completions for other contexts yet
}

const buildModuleCompletions = (moduleName: string): CompletionItem[] => {
  const functions = completions.modules[moduleName as keyof typeof completions.modules]
  if (!functions) return []

  return Object.entries(functions).map(([name, meta]) => ({
    label: name,
    kind: CompletionItemKind.Method,
    detail: `(${meta.params.join(', ')})`,
    insertText: name,
  }))
}

const buildDollarCompletions = (): CompletionItem[] => {
  return Object.entries(completions.dollar).map(([name, meta]) => ({
    label: name,
    kind: CompletionItemKind.Property,
    insertText: name,
  }))
}

Design decisions:

• CompletionItemKind.Method for module functions (shows function icon)
• CompletionItemKind.Property for dollar properties (shows property icon)
• detail field shows parameter signature
• Simple and focused - easy to extend later with descriptions

4. Server Integration

File: server/src/server.ts (updates)

Update initialization to declare trigger character:

function handleInitialize(): InitializeResult {
  connection.console.log('🦐 Server initialized with capabilities')
  return {
    capabilities: {
      textDocumentSync: TextDocumentSyncKind.Full,
      completionProvider: {
        triggerCharacters: ['.'],  // Auto-trigger on dot
      },
      semanticTokensProvider: { /* ... existing ... */ },
    },
  }
}

Update completion handler:

import { provideCompletions } from './completion/completionProvider'

function handleCompletion(params: any) {
  const document = documents.get(params.textDocument.uri)
  if (!document) return []

  const position = params.position

  // First try context-aware completions (module/dollar)
  const contextCompletions = provideCompletions(document, position)
  if (contextCompletions.length > 0) {
    return contextCompletions
  }

  // Fall back to keywords (for Ctrl+Space in general context)
  const keywords = ['if', 'else', 'do', 'end', 'and', 'or', 'true', 'false', 'null']
  return keywords.map((keyword) => ({
    label: keyword,
    kind: CompletionItemKind.Keyword,
  }))
}

Design decisions:

• Try context completions first (module/dollar)
• Fall back to keywords if no context match
• Preserves existing keyword completion behavior
• Simple, linear flow

Testing Strategy

Manual Testing

1. Type dict. → should show dict completions
2. Type list.m → should filter to map
3. Type $. → should show dollar properties
4. Type random identifier + . → should show nothing (fall back to keywords)
5. Verify signatures appear in completion detail

Future Automated Testing

• Unit tests for context analyzer with various parse trees
• Unit tests for completion provider with mock contexts
• Integration tests for end-to-end completion flow

Future Enhancements

Phase 2: Documentation

• Add description field to metadata
• Show in completion item documentation
• Consider pulling from actual prelude JSDoc comments

Phase 3: Top-Level Completions

• Show prelude globals on Ctrl+Space
• Requires different trigger logic (not just dot)
• May need position-aware filtering (don't show in strings/comments)

Phase 4: Advanced Features

• Chained completions (dict.keys.)
• Type-aware completions (know what dict.keys returns)
• Snippet completions with parameter placeholders
• Fuzzy matching for better search

References

• VSCode Language Server Guide
• CompletionItemKind Documentation
• Shrimp Grammar: src/parser/shrimp.grammar (lines 245-248 for DotGet definition)