Gunnar Morling

Thanks for the comment!

Your comment regarding DDL is interesting.

Today, this is what happens:

1/ Column doesn't exist in the destination, we'll create it based on our typing inference from the data type (important: not the data value).

2/ Certain tools will handle automatic column data type conversion if a change like this was detected at the source. We do not do this. We will simply hard fail and cause head-of-line blocking reasons being: this is anti-pattern and should be rare, in which case - it's okay to cause an err and require manual intervention for this breaking change.

3/ If the column has been dropped from the source, you as the end user can decide whether this column should be also dropped in the destination, or not. The default is not to drop it.

^ We hear more customers explicitly don't want columns to be dropped because it could cause downstream errors, such as other views / tables not compiling due to referencing a non-existent column.

We haven't heard much from folks that don't even want columns to be added. If there is a need, we can definitely add that as a config option to provide maximum configurability.

> Finally, the biggest issue with CDC always ends up being the seed loads, recoveries and the incremental snapshot strategies.

Yep totally. On the recovery bit, this is exactly why we are leveraging Kafka. If there are any particular issues, we simply don't commit the offset and cause head-of-line blocking.

On the incremental snapshot and…

If you're doing user-supplied CSVs, definitely... but if you are ingesting CSVs from a known source with known format (<insert audible sigh here>) it can definitely make sense to use a high-speed optimized ingester.

One might wonder if it might be worth the time to look into optimising the runtimes of various languages. I took a look, all operate on naive byte-by-byte scanning, and all sans PHP are written in the respective language which means any form of SIMD optimization is right off the table (okay, maybe something could be done in Java, but it seems incredibly complex, see https://www.morling.dev/blog/fizzbuzz-simd-style/):

- PHP isn't optimized anywhere, but at least it's C: https://github.com/php/php-src/blob/1c0e613cf1a24cdc159861e4406abfcee7bc1aee/ext/standard/file.c#L2051

- Python's C implementation is the same: https://github.com/python/cpython/blob/main/Modules/_csv.c

- Java doesn't have a "standard" way at all (https://www.baeldung.com/java-csv-file-array), and OpenCSV seems the usual object-oriented hell (https://sourceforge.net/p/opencsv/source/ci/master/tree/src/main/java/com/opencsv/).

- Ruby's CSV is native Ruby: https://github.com/ruby/ruby/blob/bd65757f394255ceeb2c958e87f862139dd37e7b/lib/csv.rb

Does it provide hard latency bounds like Azul does (did?), and are they lower than disk/network latencies on modern hardware?

I moved to c++/rust years ago because those languages do, and tens of milliseconds matter for network services. At the time Java could pause for 10’s of seconds, which was 1000x worse than waiting for a spinning disk to seek.

These days, disks are 100s micros to single digit millis, so I guess if Java GC is finally working 30 years after they “fixed” its performance problems, then I’d want to be able to tune ZGC to not pause the app for more than ~ 500us, max.

This article is from last year, but suggests they’re still off by an order of magnitude:

https://www.morling.dev/blog/lower-java-tail-latencies-with-zgc/

Also, that’s measuring a 30 second window.

If you hammer a 100GB-1TB heap in steady state with small allocations for, say, a month at 100% CPU, does it eventually do the typical Java thing, where a major compaction takes the process down for seconds or even minutes, or does it just slow down application requests so it can keep up with load?

Neat!

> Incidentally searching around for prior art I found this blog post https://www.morling.dev/blog/how-i-built-a-serverless-search-for-my-blog/ about building something similar using lucene, but without storing the content and only on a single lambda.

See also:

Sqlite FTS (no server): https://news.ycombinator.com/item?id=27016630

EdgeSearch (Workers): https://github.com/wilsonzlin/edgesearch

EdgeSql (Workers): https://github.com/lspgn/edge-sql

InfiniCache (Lambda): https://news.ycombinator.com/item?id=25788893

quickwit (Lambda): https://news.ycombinator.com/item?id=27074481

> I had been working on a bloom filter based index based on the ideas of bitfunnel which was developed by Bob Goodwin, Michael Hopcroft, Dan Luu, Alex Clemmer, Mihaela Curmei, Sameh Elnikety and Yuxiong He and used in Microsoft Bing https://danluu.com/bitfunnel-sigir.pdf.

Reminds me of wavelet trees: https://alexbowe.com/succinct-debruijn-graphs/ https://alexbowe.com/wavelet-trees/

Much better, because you still do have the JIT data, it is similar to having done a couple of GraalVM runs gathering profile data.

Here is an overview of using it with Maven

https://www.morling.dev/blog/building-class-data-sharing-archives-with-apache-maven/

And here is the documentation for the JIT/AOT and JIT server preview for J9, https://www.eclipse.org/openj9/docs.

By the way, this is also the way that ART evolved after the short lived AOT attempt.

Nowadays a modern version of ART will have a tiered execution of fast startup interpreter (when no native code is available), a JIT with PGO data gathering, AOT compilation from only the parts that matter when the device is idle, upload of PGO profiles into the store so that every device of similar class doesn't start from zero when an APK gets installed.

If the PGO dataset changes, then it backs into JIT again, and recompiles the native code again on idle.

I found an online version

https://icu4c-demos.unicode.org/icu-bin/collation.html

but there you first have to select the language from the drop-down menu. So in general, you would first need to know the country where your weather station was located, before you can correctly collate its name.

I don't believe the fastest entries are doing all this(?)

Edit: In the examples [1], the guy writes a Polish city using its English name "Cracow". So you can't choose the alphabetical ordering based on the geographical location of the weather station, but you need to somehow detect in which language its name is written in, in the data.

[1] https://www.morling.dev/blog/one-billion-row-challenge/

Edit2: I guess you could declare that either the "Default Unicode Collation Element Table (DUCET)", or perhaps the American English "en-US-u-va-posix" locale is the correct way to alphabetize.