Parser 2.0 (Major Delezer) #52

defunkt · 2025-12-03T00:18:58Z

defunkt commented

2025-12-03 00:18:58 +00:00

I still don't know if this is a good idea or not. But here's a new parser, written by hand instead of using lezer.

Most of the old tests pass without modification, as I've tried to match the existing APIs. If you check src/compiler/compiler.ts you can see how closely we matched it.

The only part where AI helped directly was in the string interpolation parser, so I don't take any responsibility for what's going on in there. It's possible variables/dotget won't work inside string interpolation, I still need to test it.

When fully merged, the old parser can be removed - or we can keep it for our records. We can also right away remove some limitations that lezer placed on us, such as simplifying DotGet parsing.

The only tests I'm skipping from the old suite are related to error recovery, but I'm still planning on adding that functionality to this new parser.

Along the way I found (I think) a bug where if something: won't be a function call but if something true: will, so I corrected that in the tests. Same problem with throw something vs throw something true

Also, currently DotGet does work with spaces, eg obj . prop, so we might have to explicitly disable that. I also don't think something.4.2.something works in either parser, but I believe we can make it work in the new one.

This PR also adds a new tokenizer and token-specific tests. That's where a lot of the magic happens.

On the one hand, this is nice because now we can do anything. On the other hand, it's nice to look at a grammar file to quickly get an overview of how everything is connected. In parser2, the relationships all live in code.

If you look at parser2.ts, I've written it so all the important nodes are functions you call like if() or dotGetFunctionCall(). I've also organized the parser into four sections:

constructor
important nodes, like statement() and expression()
all the other nodes in alphabetical order
helper/utility functions

I've also updated bin/shrimp so you can do bin/shrimp parse <file> or shrimp -p <file> to examine the parse tree using the new parser.

Let me know what you think!

I still don't know if this is a good idea or not. But here's a new parser, written by hand instead of using lezer. Most of the old tests pass without modification, as I've tried to match the existing APIs. If you check `src/compiler/compiler.ts` you can see how closely we matched it. The only part where AI helped directly was in the string interpolation parser, so I don't take any responsibility for what's going on in there. It's possible variables/dotget won't work inside string interpolation, I still need to test it. When fully merged, the old parser can be removed - or we can keep it for our records. We can also right away remove some limitations that lezer placed on us, such as simplifying `DotGet` parsing. The only tests I'm skipping from the old suite are related to error recovery, but I'm still planning on adding that functionality to this new parser. Along the way I found (I think) a bug where `if something:` won't be a function call but `if something true:` will, so I corrected that in the tests. Same problem with `throw something` vs `throw something true` Also, currently `DotGet` *does* work with spaces, eg `obj . prop`, so we might have to explicitly disable that. I also don't think `something.4.2.something` works in either parser, but I believe we can make it work in the new one. This PR also adds a new tokenizer and token-specific tests. That's where a lot of the magic happens. On the one hand, this is nice because now we can do anything. On the other hand, it's nice to look at a grammar file to quickly get an overview of how everything is connected. In parser2, the relationships all live in code. If you look at `parser2.ts`, I've written it so all the important nodes are functions you call like `if()` or `dotGetFunctionCall()`. I've also organized the parser into four sections: - constructor - important nodes, like `statement()` and `expression()` - all the other nodes in alphabetical order - helper/utility functions I've also updated `bin/shrimp` so you can do `bin/shrimp parse <file>` or `shrimp -p <file>` to examine the parse tree using the new parser. Let me know what you think!

defunkt added 24 commits 2025-12-03 00:18:58 +00:00

sorry lezer... 0bc923fc82

new parser(-ish) 49919e9f85

minor 109c7ff9f6

hwhitespace 1b80a159c5

try to match lezer API more closely a5b2802c0c

do allowed in arg/dict values 16cb47ddcc

make more compiler tests pass 38eaed490c

disable errors... for now! 4b6f6a127f

fix bitwise precedence 5010a9584c

regex flags, bad regexs become Words 52e100cab3

fix curly strings 1d2c85b19c

throw takes an expression c6e5c44755

fix dot.get + thing 2c0723b8a3

fix | 4b57728072

keywords are magical 4d718ef12b

use new parser in curlys de0d43a1d6

we get globals for free now c7d4db8528

switch bin/shrimp to new parser d4cb86b50b

pipes end expressions 6da00dd3c8

match lezer API ccf8f41544

you too b0ad0a0768

fix search/replace 4c91ef57b9

wrong return type 88f98f9817

bun check ee40dcc65c

defunkt added 2 commits 2025-12-03 00:49:52 +00:00

enable [ a = true ] ad044b12dc

trim keys in inspect a9df56346e

defunkt added 1 commit 2025-12-03 00:54:33 +00:00

fix ./bin/shrimp parse 3c539130b0

defunkt commented

2025-12-03 01:00:15 +00:00

One example of why we need errors back:

(new parser)

(old parser)

One example of why we need errors back: (new parser) <img width="1099" alt="image" src="/attachments/0f8bdcc2-c87e-4788-9859-f240a3198036"> (old parser) <img width="1054" alt="image" src="/attachments/fcaf5226-aa76-4b47-9261-5dc21bd7e74f">

image.png

75 KiB

image.png

74 KiB

defunkt force-pushed parser2 from 3c539130b0 to 757a50e23e

2025-12-03 01:11:47 +00:00

Compare

probablycorey referenced this pull request from probablycorey/workshop

2025-12-03 18:53:29 +00:00

Discord has a max message length of 2000, keep the PR title below that! #7

probablycorey commented

2025-12-03 19:23:01 +00:00

test, should create the thread now

_test, should create the thread now_

defunkt commented

2025-12-03 21:36:33 +00:00

Okay, compiler errors work again!

Okay, compiler errors work again! <img width="1338" alt="image" src="/attachments/1cfec4e1-2f23-4d3a-8073-8f51b93df228">

image.png

80 KiB

defunkt added 1 commit 2025-12-03 21:40:09 +00:00

restore errors, fancy printing 21e7ed41af

defunkt added 1 commit 2025-12-03 21:40:20 +00:00

no need 9b1890a3db

probablycorey approved these changes 2025-12-03 21:47:30 +00:00

probablycorey left a comment

This is slick. It is so explicit, so there is a bunch of code. But I traced through some examples and it all was pretty easy to figure out.

I can rip all the lezer and editor stuff out in another PR.

This is slick. It is so explicit, so there is a bunch of code. But I traced through some examples and it all was pretty easy to figure out. I can rip all the lezer and editor stuff out in another PR.

src/parser/node.ts

						
				@ -0,0 +88,4 @@

				  // Compound assignment operators

				  '??=': 'NullishEq',

				  '+=': 'PlusEq',

probablycorey commented

2025-12-03 21:36:27 +00:00

I've been wanting this

🎉 1

src/parser/node.ts

						
				@ -0,0 +378,4 @@

				}

				// Operator precedence (binding power) - higher = tighter binding

				export const precedence: Record<string, number> = {

probablycorey commented

2025-12-03 21:37:43 +00:00

This seems easier to understand than in the lezer file!

🚀 1

src/parser/parser2.ts

						
				@ -0,0 +59,4 @@

				      if (stmt) node.add(stmt)

				      if (this.pos === prevPos && !this.isEOF())

				        throw "parser didn't advance - you need to call next()\n\n       ${this.input}\n"