Charles Leifer

Thanks for another thing I need to add to my reading list in addition to The Art of Multiprocessor Programming.

I am really interested in parallel, asynchronous, multithreading, coroutine, futures programming so it's what I spend my days thinking about and blogging about it. I hope you sense my excitement in this comment about this topic. I'm looking for a programming model that parallelises easily and doesn't require much effort, so this PDF seems relevant to me. I really should try be a user of languages like Erlang, Inko, Pony and Go but I am too interested in the mechanism of these languages!

I am also learning from Erlang and Go, nginx and LMAX disruptor.

I don't focus on number crunching parallelisation, I let libraries and frameworks parallelise matrix multiplication such as BLAS. I'm interested in rote system parallelisation architecture. For example, PHP and nodejs is not a parallel language but how PHP is hosted in FastCGI processes means it can be executed multiple times by nginx so it is in effect parallel across requests. Unfortunately PHP and nodejs cannot create threads or use a thread pool within a request.

I want heavy CPU tasks of a request to not block other requests or the event loop and heavy IO requests to not block the event loop. I am a pre-beginner in Rust but I think you can use Rayon for CPU heavy tasks and Tokio for async IO parallelisation.

Here's a system diagram that I'm thinking about…

Charles provided really great documentation on how to build this here: http://charlesleifer.com/blog/compiling-sqlite-for-use-with-python-applications/

Or if you're feeling lazy (like I was), there's a fork of his library at https://github.com/karlb/pysqlite3 which compiles the 3.25.0 amalgamation by default. This worked for me:

$ python3 -mvirtualenv venv
$ source venv/bin/activate
$ pip install git+git://github.com/karlb/pysqlite3
Collecting git+git://github.com/karlb/pysqlite3
...
Installing collected packages: pysqlite3
Successfully installed pysqlite3-0.2.0
$ python
Python 3.6.5 (default, Mar 30 2018, 06:41:53)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pysqlite3
>>> pysqlite3.connect(":memory:").execute("select sqlite_version()").fetchall()
[('3.25.0',)]

Loved Brooks’ and feel like I must’ve internalized his model because that’s all I care about anymore in dev. Coherence among the components (people or software) to produce something good. I think of if 100 pianos are tuned to the same tuning fork then they all play well together.

AI accelerated iteration and undisciplined use seem to me the main risk lately. Drift from any prompting, accumulation of features, plausible looking patches that are actually slop. Wrote about this recently [0], it creates a dynamic where you aren’t just making the software less coherent, you’re paying more and more to make it so. Curious to hear others experience of this.

[0] https://charlesleifer.com/blog/tokens-and-dreams/

Vedis is called an ACID database on its website and rlite is called "transactional" in the read-me, so I assume that, like SQLite, by default they sync at the end of each transaction. Even if this weren't the case, though, I think the comparison would still be appropriate. Any embedded database, regardless of its stated use case, is in direct competition with SQLite because the latter dominates the market (and for good reason!).

Looking for existing benchmarks of this kind I have found one that compares Vedis, Redis and SQLite: http://charlesleifer.com/blog/completely-un-scientific-benchmarks-of-some-embedded-databases-with-python/.

Hi @coleifer, I'll disagree, I'm the author. I don't think they are gross hacks, just a more sensible default configuration. In fact, I used a lot of your advice from your blog post https://charlesleifer.com/blog/going-fast-with-sqlite-and-python/ while making this library.

Fair enough if you want to compile your own version, but there's a lot of people and Python devs who don't know how / won't go through the effort and that's what this project is for.

It's also a work in progress and not meant to be released yet. Shoot me an e-mail ajay@plasticityai.com if you have suggestions for how to improve it.

This might benefit from SQLite's Virtual tables: https://sqlite.org/vtab.html

With Virtual Tables you can expose any data source as a SQLite table -- then you can use every SQL feature that sqlite offers. You can just tell sqlite how to iterate through your data with a few functions, with an option to push down filtering information for efficiency.

You can also create your own aggregates, functions etc.

Here's an article where the author exposes redis as a table within sqlite: http://charlesleifer.com/blog/extending-sqlite-with-python/