Software Engineering Daily

I think the reason this is discussed now is because yesterday I tried to re-submit the Apache Arrow article. Here's what I wrote: https://en.wikipedia.org/w/index.php?title=Draft:Apache_Arro... It was rejected / reverted 10 minutes later by a Wikipedia editor. The blog post from Justin was in July 2019 (https://www.dremio.com/why-apache-arrow-wikipedia/)

There's many interesting and good points in the discussion here, thank you!

To add my 2 cents:

- Apache Arrow is notable, deserves a Wikipedia page. It might not have been when someone first tried to create a Wikipedia page for it in 2017 (see https://en.wikipedia.org/w/index.php?title=Draft:Apache_Arro...), but in the three years since it has become a major project, see e.g. https://blogs.apache.org/foundation/entry/the-apache-softwar... Notability is clearly subjective, depends on what the author and reviewer find interesting. In the variant I submitted yesterday I tried to make it clear why it's notable - Apache arrow is a standard format that connects different languages, runtimes, data systems, communities, e.g. the Python and Java data communities. See e.g. https://wesmckinney.com/blog/apache-arrow-pandas-internals/ - Apache Arrow is to my knowledge partly the brainchild of Wes McKinney, creator of pandas, it's his attempt (looking strongly like success) to resolve a major issue in data science. - I think it's a good point Justin made at https://www.dremio.com/why-apache-arrow-wikipedia/ that it's bad that Wikipedia editors reject articles on stuff they know nothing about - if you look at their profiles, they don't seem to have any knowledge or interest about technology or software. That's not a good system. - I haven't contributed to Wikipedia really before, and I don't understand the rules, I admit that. Probably what I did yesterday was just not following their process, and that's the reason my edit was reverted. I guess it's also true that Justin at first didn't do a great job at submitting an impartial, non-PR article. However, my understanding from looking at some drafts and the talk page is that he then took the editor comments into account, and the last variant of the page he tried to submit in July 2019 was OK. - So overall I think the answer to the question "Why isn't there a Wikipedia page on Apache arrow?" is that it's an unfortunate case of authors and editors not doing a great job. At least I'm pretty sure I didn't do a good job yesterday, I wanted to help, but only had an hour, not a day to learn how Wikipedia ticks and to do more research to find better references. I hope someone with more experience in Wikipedia and Arrow will try to re-write and re-submit the Wikipedia article in the future. - The rule to discourage (or forbid?) people involved with Apache Arrow from contributing to its Wikipedia page is unfortunate. I recently started to use it and learn about it, but I don't know much about it at this point. E.g. Wes McKinney has written at this point 8 high-quality blog posts about it (https://wesmckinney.com/archives.html) - those don't count as references? Even if he or the Apache Arrow team wrote a paper about it, it wouldn't count because it's a primary source, and Wikipedia only wants secondary sources to establish notability? There are ~ 100 videos on YouTube, and many blog posts and a few podcasts (e.g. https://softwareengineeringdaily.com/2016/07/17/apache-arrow...) that mention Apache Arrow. Naturally almost all of them are from Apache Arrow contributors, or from companies using Apache Arrow. - Apache Arrow has an interesting story, and it has evolved over the past years and will keep evolving, so I think exactly for that reason a Wikipedia page would be good to have, since the current project page and old blog posts don't capture that well.

Everyone else here has covered explicit file naming and data structure. That's totally key, but I'm trying to do what you're doing with Keybase[00] right now. Migrating away from Dropbox, Google Drive/Docs, etc.

A caveat is that files are limited to 250Gb currently and private repos are 100Gb personal and 100gb per team. I'm not sure on team creation limits and I don't think they are offering paid storage yet. Because of this, I'm also looking at IPFS[0] for cold storage and using other methods to catalog and index.

They added encrypted private git[1] registries and dropbox like file management[2]. That plus the mobile app, teams (groups), chat, and that it's primarily an identity provider I think it's a killer platform for file management and organization. It's still kind of immature in the chat and friends list area (not in a way that makes it unusable, just a little unpolished), but it's open source[3] and all of the functionality that is there works flawlessly for me so far.

Their recent edition of "bomb" messages[4], where you can send a private message with an expiration, I've found to be super useful. It has clients for all major operating systems and mobile platforms. Source is right there on github if you'd like to modify/enhance. You can even browse your git repos from your phone! They haven't added the ability to commit/push to git from mobile yet.

Now I don't have to worry that if I change a credit card and forget to update dropbox that I'll loose my stuff. Which happened this year. I didn't loose anything, but I realized I had not checked in on my over a decade worth of archives in a bit and hadn't paid them.

I took a look at Keybase a couple months ago and decided to make this move after hearing the Software Engineering Daily podcast with the found Max Krohn[5]. It's a pretty good walk through on their goals and current offerings.

Even with it's lack of commercial SaaS flair, I've actually found using it for chats, file sharing, git, and organization to be a fairly large productivity boost. Maybe because it's pretty no nonsense. I've moved half of my work dev chat to it from slack, and I don't miss Slack at all except for video chat/screen share. I especially don't miss it's the terrible file management / organization.

The next phase of my organization, I'm working on connecting it to a personal web portal where I can share and access control the content, but also curate my links, writing, and media I create in an HN/reddit style feed and use Kb to federate to other people's (friends) sites for content to and group that content by relationship and content category. So basically a smarter web ring (Everything old is new again!) with RBAC/circle of trust access control and collaborative versioned content editing.

I'm also exploring other similar tech in tandem or instead of Keybase like IPFS/Mastodon/WebTorrent. So far, for a lot of what I want though, I think Kb has it nicely tied up with a bow, though I doubt it'll scale to any interesting level of traffic without some elbow grease on the ops/storage side.

[00] https://keybase.io

[0] https://ipfs.io/

[1] https://keybase.io/blog/encrypted-git-for-everyone

[2] https://keybase.io/docs/kbfs

[3] https://github.com/keybase

[4] https://keybase.io/blog/keybase-exploding-messages

[5] https://softwareengineeringdaily.com/2017/10/24/keybase-with...

It's really cool to see these techniques in the wild, and also feels encouraging to us as we're doing something very similar at Splitgraph [0] to implement our "Data Delivery Network" [1]. Recently we've started calling Splitgraph a "Data Mesh" [2]. As long as we have a plugin [3] for a data source, users can connect external data sources to Splitgraph and make them addressable alongside all the other data on the platform, including versioned snapshots of data called data images. [4] So you can `SELECT FROM namespace/repo:tag` where `tag` can refer to an immutable version of the data, or e.g. `live` to route to route to a live external data source via FDW. So far we have plugins for Snowflake, CSV in S3 buckets, MongoDB, ElasticSearch, Postgres, and a few others, like Socrata data portals (which we use to index 40k open public datasets).

Our goal with Splitgraph is to provide a single interface to query and discover data. Our product integrates the discovery layer (a data catalog) with the query layer (a Postgres compatible proxy to data sources, aka a "data mesh" or perhaps "data lake"). This way, we improve both the catalog and the access layer in ways that would be difficult or impossible as separate products. The catalog can index live data without "drift" problems. And since the query layer is a Postgres-compatible proxy, we can apply data governance rules at query time that the user defines in the web catalog (e.g. sharing data, access control, column masking, query whitelisting, rewriting, rate limiting, auditing, firewalling, etc.).

We like to use GitLab's strategy as an analogy. GitLab may not have the best CI, the best source control, the best Kubernetes deploy orchestration, but by integrating them all together in one platform, they have a multiplicative effect on the platform itself. We think the same logic can apply to the data stack. In our vision of the world, a "data mesh" integrated with a "data catalog" can augment or eventually replace various complicated ETL and warehousing workflows.

P.S. We're hiring immediately for all-remote Senior Software Engineer positions, frontend and backend [5].

P.P.S. We also have a private beta program where we can deploy a full Splitgraph stack onto either self-hosted or managed infrastructure. If you want that, get in touch. We'll probably be in beta for 12-18 months.

[0] https://www.splitgraph.com

[1] We talked about all this in depth on a podcast: https://softwareengineeringdaily.com/2020/11/06/splitgraph-d...

[2] https://martinfowler.com/articles/data-monolith-to-mesh.html

[3] https://www.splitgraph.com/blog/foreign-data-wrappers

[4] https://www.splitgraph.com/docs/concepts/images

[5] Job posting: https://www.notion.so/splitgraph/Splitgraph-is-Hiring-25b421...

I develop software for a living, and I dare say I'm better than most at it. I've been doing it about 20 years, since I was writing Z80 assembly as a middle schooler [1]. It would take a lot more than a year for someone to learn to do my job. But I'm a lead engineer, not an entry level developer.

However, it would take less than the total amount of time I've been building software to do what I do, because my path to my job took me through all sorts of software development I don't do anymore. That includes 8 years of higher ed, of which I apply only a tiny fraction on a daily basis. Don't get me wrong, I wouldn't trade all that education for the world. I draw upon it in plenty of indirect ways. I just know there are more direct routes for someone who simply wants to become an application developer.

I know this, because I know plenty of people who have taken less than a year to get into the industry, and are now productive developers. One of my friends even wrote about how he did it in detail [2]. There's a fantastic episode of Software Engineering Daily interviewing a guy who landed a lucrative job at AirBnB little more than a year after doing a dev bootcamp [3]. My own brother has been in the industry less than a year after coming through General Assembly and is doing quite well. I interview people all the time who have similar stories. They're not unicorns.

Developing software isn't easy. But it's also not that hard. At least the sort of software most of us build. You only need a couple people at the typical enterprise who are capable of doing architecture and the most technically intense stuff.

[1] http://www.ticalc.org/archives/files/fileinfo/61/6175.html

[2] http://alexkrupp.typepad.com/sensemaking/2013/11/2012-my-yea...

[3] http://softwareengineeringdaily.com/2016/07/11/salary-negoti...

If I understand correctly you are talking about stuff like a debugger for an opaque wasm binary. The kind of tooling a system administrator could want to manage various wasm programs running on their servers.

I will assume so :)

There was a nice article detailing how for many application usage wasm today is less safe (under certain aspects) of x86, specifically due to the high quality level of tooling available for x86 and due to the fact that today debugging/inspection support of wasm is very low.

Other than that I would say that it is specific to the constraints you have in a browser. Two of which are the ability to run on minimal implementations requirement (as new optional features might be missing and might be running in a variety of environments) and the ability to safely run arbitrary untrusted code.

If you do not care about these then wasm does not offer much to you.

On the other hand it is perfect as a compilation target for plugins.

> application developers won't be touching it directly.

On the web it is meant to replace asm.js, it is not meant to be written by hand (unless you are a compiler developer).

The whole point of the article in this case is that it is a perfect interface for plugins over a specific and limited interface that allows running untrusted and opaque binaries.

For this specific usage only high-level languages where a viable choice up to now (two common choices were js and lua). wasm is comparatively closer to C than javascript/lua, but it is still safe to arbitrarily embed without using OS-level mechanisms.

It is a very specific use case, but also a relatively common one in applications.

One of the big sponsors (Fastly) in investing a lot in it due to the usefulness for them regarding lightweight sandboxing (for this reason they are also developing a debugger for wasm [1]).

On the other hand it is essentially useless if your application is something like a unix utility.

[1] I am not sure it is in this episode, but here the CTO of fastly explain why it is useful for their use case: https://softwareengineeringdaily.com/2019/09/25/webassembly-...

I only found mention of v8-isolate when searching for it specifically. Coming up in only https://blog.cloudflare.com/serverless-performance-compariso... and a few comments on HN. I don't see any description of its use other than a summary of 'we use v8 isolates', which begs a /r/restofthefuckingowl. Even an interview with KV doesn't reveal much. https://softwareengineeringdaily.com/wp-content/uploads/2018... Isolate also doesn't do anything for sandboxing from what I can tell. Cloudflare quotes 'That said, we have added additional layers of our own sandboxing on top of V8.' I do not see any actual technical information on how they're using v8.

v8 and by extension isolated-vm by itself does not cover things like fetch if we're talking JS. https://github.com/laverdet/isolated-vm/issues/63 This is an immediete killer as js browser features are part of why a js engine is wanted.

Having node handling these things is a time bomb of debugging where instead of debugging v8 you're debugging node. There are node version problems with stability as mentioned on the isolate-vm readme.

DOS goes beyond just CPU time and memory. Wasm is being treated as an environment of its own with some lower level accessibility. Unless the API allows feature whitelisting - I can't find any docs relating to that - then I don't see anything stopping thread creation. I'd be interested to know how cloudflare prevents these features. isolate-vm doesn't appear to interact with the v8 api with options relating to this.

Fly.io has their fly engine meant for local development so you can deploy onto their servers. https://github.com/superfly/fly