Federico Tomassetti

Super cool. Bidirectional code generation is something I've spent a bit of time thinking about: I've been building a spreadsheet that generates Python code when you edit it [1], but some of our users also want the ability to edit the Python code they generate and have that reflect in the sheet itself.

Template -> Code -> Template is one really hard part of this, and something this tool seems to take a really good approach to. If you're interested in related subjects (like transpiliation), I'd recommend this overview article as a great approach [2]. From this article: "Now, there is a single biggest mistake we see in persons trying to implement a transpiler without experience in this: they try to generate directly the code of the target language from the AST of the original language."

Question to the OP - you mention you parse the Ruby AST - do you also transform this into an AST of your template syntax before generating the template? Aka, do you avoid this sin, or is it not an issue for you?

With Mito, there is additional complexity beyond just going from Template -> Code -> Template, in that we also need to understand _which_ variables are being changed and in what way. This is necessary because a spreadsheet stores other data about your variables beyond just their current value. As an example, which of these columns in a dataframe are a result of a formula vs. being in the original dataset isn't something that is just stored in the dataframe itself.

I haven't tried too…

I know of no Regex pattern that can handle all the old and new HTML as well as HTML5: believe me, as one who is looking to put HTML parser on FPGA/ASIC for higher speed, I've actually forayed down this rabbit hole a few times in the fruitless pursuit of identifying this elusive pure Regex pattern for HTML, et. al. Problem is in Regex's lack of support for multiple state machine and its needed interactions between these state machines.

The language Perl came closest to the smallest HTML parser.

Things to do before doing simplistic regex on HTML using some multiple passes of Regex are probably required, probably in order of (my 20yo memory failing here):

- de-CDATA

- De-pairing of quotes

- De-symbolization of HTML symbols, entities. and codes (de-escaping)

- lone unterminated </> (ie. <p>)

Before you can even hit up for pairing of <XXX> and </XXX> and getting to its HTML tags and attributes.

In short, additional scripting is required to conduct the applying of multiple Regex patterns before one can even be getting into properly parsing the HTML.

Simplest that I've gotten is using both bash logic and Regex, but it fails on certain HTML codes.

Federico Tommassetti, well-renown expert on domain specific languages and transpiliers, covers nearly all the valid libraries of many modern languages for just the parsing of HTML.

Federico makes it easier for first timer of HTML parser coding to that that first step: selecting an HTML parser…

You could take this idea further by fully parsing queries according to the grammar of the SQL language, rather than using simple pattern matching.

In fact it is a result of theoretical computer science that you _cannot_ correctly parse languages like SQL, HTML, Python, etc. with regular expressions: Any attempt to do anything non-trivial will have cases where it misunderstands the code.

So you would want to find a SQL grammar (an outdated example in [1]) and a module[2] that can use this to parse queries into a data structure to which you can apply transformations (e.g., changing case of keyword tokens) and then write back out as a string.

SQLite's documentation has some nice diagrams[3] to get an idea for how it parses a query string. The table of links at the top lets you dive into, e.g., all the optional parts of a SELECT statement.

1: https://ronsavage.github.io/SQL/sql-92.bnf.html

2: https://tomassetti.me/parsing-in-python/

3: https://sqlite.org/lang.html

From what I can tell, your best bet would be to build the generated parser (which will generate some C code) and then build a small interface in C. From there I would just use R/Python's foreign function interface to call your C wrapper.

Alternatively for python you could use a parse combinator like parsec. Depending on how complicated your file can get, using parsec could very likely be the better choice as it has first class support in python and seems powerful enough to handle your use case.

There do seem to be a number of parse combinators in R but none of them seem as well established as parsec for python (which is based on the much more well known parsec for haskell).

A quick look around seems to show that python will be your best bet for parsing as it has some decent tooling. Here is an article on some of the different parse generators/parse combinators in the space.

https://tomassetti.me/parsing-in-python/

Somewhat similar, see https://substrait.io/

So for example using DuckDB with the Substrait extension, if you create a table

create table t(a int); and then query it as in the article, you can see something similar to what is described in the article

CALL get_substrait_json('select * from t'); {"relations":[{"root":{"input":{"project":{"input":{"read":{"baseSchema":{"names":["a"],"struct":{"types":[{"i32":{"nullability":"NULLABILITY_NULLABLE"}}],... DuckDB extension doesn't seem to cover any DDL operations though.

https://duckdb.org/docs/extensions/substrait

Some other related discussions and links that i've collected over the years

https://news.ycombinator.com/item?id=37415494

https://news.ycombinator.com/item?id=34233697

https://news.ycombinator.com/item?id=31981568

https://datastation.multiprocess.io/blog/2022-04-11-sql-parsers.html

https://tomassetti.me/parsing-sql/

https://tomassetti.me/recognizing-hand-written-shapes-programmatically-find-the-key-points-of-a-rectangle/

https://azure.microsoft.com/fr-fr/blog/turn-your-whiteboard-sketches-to-working-code-in-seconds-with-sketch2code/

https://www.theverge.com/2017/4/11/15263434/google-ai-autodraw-doodle-bot-drawing-image-recognition

https://quickdraw.withgoogle.com/#