Status
- (In progress) TurboIntegrator reserved words
- (Completed) ASCII and Text TurboIntegrator Functions
- (In progress) Attribute Manipulation TurboIntegrator Functions
- Chore Management TurboIntegrator Functions
- Cube Manipulation TurboIntegrator Functions
- Data Reservation TurboIntegrator Functions
- Date and Time TurboIntegrator Functions
- Dimension Manipulation TurboIntegrator Functions
- Hierarchy Manipulation TurboIntegrator Functions
- ODBC TurboIntegrator Functions
- Process Control TurboIntegrator Functions
- Rules Management TurboIntegrator Functions
- Sandbox Functions
- Security TurboIntegrator Functions
- Server Manipulation TurboIntegrator Functions
- Subset Manipulation TurboIntegrator Functions
- View Manipulation TurboIntegrator Functions
- Miscellaneous TurboIntegrator Functions
I've begun work on building a language grammar for TurboIntegrator processes. The project (linked at the top) is a bit slow going due to how TI syntax works, but it's coming along. Unlike other languages, there's not a whole lot of flexibility when it comes to TI logic, meaning that much of the language grammar is matching syntax for builtin functions. This isn't terrible, but it does mean that there's a lot of room to group and condense rules, which is a very iterative process. It also means that this will probably be one of the larger tree-sitter grammars, but I'm hoping over time things could be grouped more efficiently.
Support is limited to some text functions and a few attribute manipulation functions, but I'm adding more pretty frequently. To give an example of what it looks like, let's take a dummy text file (test.pro) with some valid TI code:
test.pro
Code: Select all
# This is a comment
asciidelete('myfile.txt');
asciioutput("my_file.txt", 'abc', v1);
asciioutputopen("my_file.txt", FILE_OPEN_APPEND());
numbertostring(12345.123);
numbertostringex(12345.123, "abc", ".", ',');
setinputcharacterset("TM1CS_UTF8");
setoutputcharacterset('file_name.txt', "TM1CS_UTF8");
setoutputescapedoublequote("my_file.txt", 1);
stringtonumber('12345.57');
stringtonumberex('12345.57', ".", '.');
textoutput('my_fil123.txt', "myVar", var1, var2);
### This is another comment
attrnl('Model', "abc", "size", "en");
attrsl('Model', "efg", "size", "en");
attrdelete("dimension", 'name');
attrinsert('Model', "Transmission", 'InteriorColor', 'N');
attrputn(123.45, "Dimension", "ElName", 'AttrName', "ProdCode");
attrputs("blah", "Dimension", "ElName", 'AttrName', "ProdCode");
Code: Select all
(source_file [0, 0] - [19, 0]
(comment [0, 0] - [0, 19])
(builtin_funcs [1, 0] - [1, 26]
(ascii_and_text_funcs [1, 0] - [1, 25]
(asciidelete [1, 0] - [1, 25]
(quoted_string [1, 12] - [1, 24]))))
(builtin_funcs [2, 0] - [2, 38]
(ascii_and_text_funcs [2, 0] - [2, 37]
(ascii_or_text_output [2, 0] - [2, 37]
(quoted_string [2, 12] - [2, 25])
(quoted_string [2, 27] - [2, 32])
(word [2, 34] - [2, 36]))))
(builtin_funcs [3, 0] - [3, 51]
(ascii_and_text_funcs [3, 0] - [3, 50]
(asciioutputopen [3, 0] - [3, 50]
(quoted_string [3, 16] - [3, 29]))))
(builtin_funcs [4, 0] - [4, 26]
(ascii_and_text_funcs [4, 0] - [4, 25]
(numbertostring [4, 0] - [4, 25]
(float [4, 15] - [4, 24]))))
(builtin_funcs [5, 0] - [5, 45]
(ascii_and_text_funcs [5, 0] - [5, 44]
(numbertostringex [5, 0] - [5, 44]
(float [5, 17] - [5, 26])
(quoted_string [5, 28] - [5, 33])
(num_separator [5, 35] - [5, 38])
(num_separator [5, 40] - [5, 43]))))
(builtin_funcs [6, 0] - [6, 35]
(ascii_and_text_funcs [6, 0] - [6, 34]
(setinputcharacterset [6, 0] - [6, 34]
(char_set [6, 22] - [6, 32]))))
(builtin_funcs [7, 0] - [7, 53]
(ascii_and_text_funcs [7, 0] - [7, 52]
(setoutputcharacterset [7, 0] - [7, 52]
(quoted_string [7, 22] - [7, 37])
(char_set [7, 40] - [7, 50]))))
(builtin_funcs [8, 0] - [8, 45]
(ascii_and_text_funcs [8, 0] - [8, 44]
(setoutputescapedoublequote [8, 0] - [8, 44]
(quoted_string [8, 27] - [8, 40]))))
(builtin_funcs [9, 0] - [9, 27]
(ascii_and_text_funcs [9, 0] - [9, 26]
(stringtonumber [9, 0] - [9, 26]
(quoted_string [9, 15] - [9, 25]))))
(builtin_funcs [10, 0] - [10, 39]
(ascii_and_text_funcs [10, 0] - [10, 38]
(stringtonumberex [10, 0] - [10, 38]
(quoted_string [10, 17] - [10, 27])
(num_separator [10, 29] - [10, 32])
(num_separator [10, 34] - [10, 37]))))
(builtin_funcs [11, 0] - [11, 49]
(ascii_and_text_funcs [11, 0] - [11, 48]
(ascii_or_text_output [11, 0] - [11, 48]
(quoted_string [11, 11] - [11, 26])
(quoted_string [11, 28] - [11, 35])
(word [11, 37] - [11, 41])
(word [11, 43] - [11, 47]))))
(comment [12, 0] - [12, 27])
(builtin_funcs [13, 0] - [13, 37]
(attr_funcs [13, 0] - [13, 36]
(attrnl_and_attrsl [13, 0] - [13, 36]
(quoted_string [13, 7] - [13, 14])
(quoted_string [13, 16] - [13, 21])
(quoted_string [13, 23] - [13, 29])
(quoted_string [13, 31] - [13, 35]))))
(builtin_funcs [14, 0] - [14, 37]
(attr_funcs [14, 0] - [14, 36]
(attrnl_and_attrsl [14, 0] - [14, 36]
(quoted_string [14, 7] - [14, 14])
(quoted_string [14, 16] - [14, 21])
(quoted_string [14, 23] - [14, 29])
(quoted_string [14, 31] - [14, 35]))))
(builtin_funcs [15, 0] - [15, 32]
(attr_funcs [15, 0] - [15, 31]
(attrdelete [15, 0] - [15, 31]
(quoted_string [15, 11] - [15, 22])
(quoted_string [15, 24] - [15, 30]))))
(builtin_funcs [16, 0] - [16, 58]
(attr_funcs [16, 0] - [16, 57]
(attrinsert [16, 0] - [16, 57]
(quoted_string [16, 11] - [16, 18])
(quoted_string [16, 20] - [16, 34])
(quoted_string [16, 36] - [16, 51])
(attr_type [16, 54] - [16, 55]))))
(builtin_funcs [17, 0] - [17, 64]
(attr_funcs [17, 0] - [17, 63]
(attrputn_and_attrputs [17, 0] - [17, 63]
(float [17, 9] - [17, 15])
(quoted_string [17, 17] - [17, 28])
(quoted_string [17, 30] - [17, 38])
(quoted_string [17, 40] - [17, 50])
(quoted_string [17, 52] - [17, 62]))))
(builtin_funcs [18, 0] - [18, 64]
(attr_funcs [18, 0] - [18, 63]
(attrputn_and_attrputs [18, 0] - [18, 63]
(quoted_string [18, 9] - [18, 15])
(quoted_string [18, 17] - [18, 28])
(quoted_string [18, 30] - [18, 38])
(quoted_string [18, 40] - [18, 50])
(quoted_string [18, 52] - [18, 62])))))
Running our source file through the command line syntax highlighting utility produces the following (pasted as an image):
Just to reiterate the flexibility of having a syntax tree, I produced a highlighted version of my code independent of an editor, in my terminal.
Tree-sitter can also report errors in syntax. Let's assume we forgot a semicolon on the last statement of test.pro. The syntax tree would display the following:
Code: Select all
...
(builtin_funcs [18, 0] - [18, 63]
(attr_funcs [18, 0] - [18, 63]
(attrputn_and_attrputs [18, 0] - [18, 63]
(quoted_string [18, 9] - [18, 15])
(quoted_string [18, 17] - [18, 28])
(quoted_string [18, 30] - [18, 38])
(quoted_string [18, 40] - [18, 50])
(quoted_string [18, 52] - [18, 62])))))
test.pro 0.12 ms 6474 bytes/ms (MISSING ";" [18, 63] - [18, 63])
That pretty much sums it up! In theory this could enable a lot more than just syntax highlighting in both the web and various applications that support tree-sitter. Each editor will have its own way of loading the language grammar (I personally use Emacs for, well, everything), so you'll need to consult that specific documentation on a case-by-case basis.
If you want to contribute, please send patches using the git-send-email workflow to dev@mailcd.com. Last note, this is licensed under the 3-clause BSD license, so feel free to use it, fork it, extend it, or sell it.
---
Additional Details:
High-level the tree-sitter process works like this:
- A parser generator is provided a language grammar
- A language parser is created based on the language grammar
- The parser is used to build a concrete syntax tree (CST) of a syntax file
- The CST is paired with highlighting rules to match syntax nodes with highlighting rules (colors, etc.)
There are multiple parser generator projects, but tree-sitter seems to be gaining wide adoption and is integrating into many larger projects. Tree-sitter can also be used in both applications and the browser, making it extremely versatile. Tree-sitter by itself just produces the CST (and may provide highlighting groups), but how the CST is used is completely open-ended. Although commonly used for syntax highlighting, it can also be used for things such as code folding, syntax-based error reporting, and other useful features.
Here's a list of a few places where tree-sitter is currently being used (either directly or through a plugin):
- Emacs (as of version 29)
- Atom
- VSCode
- GitHub (used in the browser)
- Probably more
If you're interested in finding out more, I'd recommend starting with the tree-sitter documentation: https://tree-sitter.github.io/tree-sitter/