It's a tree...

Finally have the lexer to tree functionality going... yes, I took a year off. Looking very nice!

This is a single-pass dictionary-based lexing system. The lexer creates a stack of matches and then commits the stack into the tree as child nodes. Recursive parsing yanks out code blocks and adds them to the parent's node. Right now we're only scanning import, class and method blocks.

I've also expanded the dictionary system to be able to use sub-dictionaries. This will make expression parsing faster since it doesn't need to look for parse rules which don't apply.

Anyway, sample scan below:

Scan complete: /Users/rdavenpo/git/plato/plato_parser/buffer.plato
0-Node 0:SCAN_IMPORT [import]
1-Node 0:SCAN_TERM [system.libc]
0-Node 0:SCAN_IMPORT [import]
1-Node 0:SCAN_TERM [std.Math]
0-Node 0:SCAN_METHOD [(null)]
1-Node 0:SCAN_METHOD [static]
2-Node 0:SCAN_METHOD [int]
3-Node 0:SCAN_METHOD [_cdecl]
4-Node 0:SCAN_METHOD [main]
5-Node 0:SCAN_NONE [{
printf("Hello, World!\n");
}]
0-Node 0:SCAN_CLASS [class]
1-Node 0:SCAN_STORAGE [Buffer]
2-Node 0:SCAN_NONE [(null)]
1-Node 0:SCAN_METHOD [(null)]
2-Node 0:SCAN_METHOD [Buffer]
3-Node 0:SCAN_NONE [{
buffer = malloc(size);
if(buffer == null){
Err.Warning("Out of memory");
}
printf("// Hello my friend!\n");
length=0;
bufsize = size;
return;
}]
1-Node 0:SCAN_METHOD [(null)]
2-Node 0:SCAN_METHOD [public]
3-Node 0:SCAN_METHOD [Set]
4-Node 0:SCAN_NONE [{
memset(buffer, set, bufsize);
return;
}]
1-Node 0:SCAN_METHOD [(null)]
2-Node 0:SCAN_METHOD [public]
3-Node 0:SCAN_METHOD [Zero]
4-Node 0:SCAN_NONE [{
Set(0);
}]
1-Node 0:SCAN_METHOD [(null)]
2-Node 0:SCAN_METHOD [public]
3-Node 0:SCAN_METHOD [Buffer]
4-Node 0:SCAN_METHOD [Copy]
5-Node 0:SCAN_NONE [{
Buffer newBuf = new Buffer(bufsize);
this.CopyTo(newBuf, 0, newBuf.bufsize);
return newBuf;
}]
1-Node 0:SCAN_METHOD [(null)]
2-Node 0:SCAN_METHOD [public]
3-Node 0:SCAN_METHOD [CopyTo]
4-Node 0:SCAN_NONE [{
memcpy(target.buffer, buffer+targetStart, targetLength);
}]
0-Node 0:SCAN_METHOD [(null)]
1-Node 0:SCAN_METHOD [int]
2-Node 0:SCAN_METHOD [end]
3-Node 0:SCAN_NONE [{
printf("Goodbyte, World!\n");
}]