GoogleFOSSSydney
13 HN recommenders · recommended in 17 months, 2012–2025
51:18 · 162K views · Aug 30, 2011
Rob Pike shows how the Go text/template lexer is built, using state functions, goroutines and channels. The talk was given at a Google Technology User Group in Sydney in 2011.
HN returns to it for years as an elegant introduction to writing a lexer that is easy to follow even without a parsing background.
20 comments from Hacker News
Lexical Scanning in Go by Rob Pike this video I love everything about this talk. It walks you through building a lexer from scratch in a simple and elegant way, through a very interesting use of coroutines. I appreciate the bits of humor in the talk as well.
Rob Pike did a really good talk entitled 'Lexical Scanning In Go'[0], which covers how the lexing for text/template works. It's far more interesting than it sounds, and even if you don't know much about lexing/parsing it's easy to follow. [0] this video
Personally, I have a penchant for writing my own pull parsers. Its a mind-expanding exercise. The neat thing about Go is that parsers can return functions that consume the next token. Rob Pike has an excellent video about this: this video
Related, I love Rob Pike's talk about lexical Scanning in Go (2011). Educational and elegant approach. this video
I would like to mention that most, if not all, of the Go compiler and related utilities is written in Go itself, with high quality, open source code that can be found in the "go" package of the standard library: [LINK] I would recommend using that as a template for one's own compiler, because it's written in idiomatic Go (originally by the language creators) and is cleanly split across abstraction boundaries (packages "scanner", "parser", "ast", and so on.) There's also a smaller, simpler parser in the "text/template/parse" package, that is used to parse the templating languages in "text/template" and "html/template". As an introduction to it, I recommend the talk "Lexical Scanning in Go" by Rob Pike: this video