Superpowered Blog

Meanwhile, Android Chrome deliberately introduces a ~300 millisecond audio delay to make web games impossible and force game devs to use the Play Store so that Google gets its pound of flesh. The superpowered[1] database has many measurements showing how bad Android is.

Those of you with Android devices can try this ancient web audio demo to see that the issue remains unfixed all these years later, except on certain select premium devices (certain Samsungs, etc.).

https://webaudiodemos.appspot.com/TouchPad/index.html

You should hear sound the instant a rectangle is touched/clicked and you will with a desktop browser but not with most Androids.

Apple also employs dark patterns on iOS Safari to make web games impossible:

1. No full screen. ("But the user wouldn't know how to exit full-screen - unless they buy from the App Store! We also plan to make full screen PWAs impossible because of, like, Reasons and totally not because of our 30% App Store cut!")

2. Slow WebGL and no WebGL 2.0. ("But we need to run extra shader validation security checks every single frame even though we technically only need to do it once up front! But this extra Privacy and Security[TM] is totally unnecessary if you buy from the App Store!")

3. No device motion tilt control. ("Allowing the user to consent to tilt control would violate the user's Privacy and Security[TM] - unless they buy from the App Store!")

4. No JS debug console unless you register an Apple developer account, download 10+…

Sure.

Originally Android only had the Java media APIs, with a quite complex state.

https://developer.android.com/reference/android/media/MediaPlayer.html

Android 2.3 introduced OpenSL ES into the NDK and Android 4.0 got OpenMAX.

All of them weren't up to the game of realtime audio due to the Android architecture.

http://superpowered.com/androidaudiopathlatency

https://nsr.cse.buffalo.edu/wp-content/uploads/2016/10/rtdroid-sound-jtres16.pdf

https://www.androidauthority.com/android-6-0-marshmallow-audio-latency-670727/

There are plenty of other articles online when you search for audio issues on Android.

There was even a Google IO 2013 talk about it.

https://www.youtube.com/watch?v=d3kfEeMZ65c

Meanwhile Samsung had customized their Android versions to properly support real time audio and offered an SDK for their handsets.

http://developer.samsung.com/galaxy/professional-audio

At Google IO 2016 the problem was again discussed, with the Samsung team from Professional Audio being called on stage to discuss their collaboration.

https://www.youtube.com/watch?v=F2ZDp-eNrh4

Finally last year at Google IO 2017, the new real time audio was announced, as C only lib.

https://www.youtube.com/watch?v=C0BPXZIvG-Q

https://developer.android.com/ndk/guides/audio/index.html

The announced C++ wrapper ended up being an external project on github, detached from the NDK, that you need to integrate yourself, without any guarantees how long Google will care to maintain it.

Oh, to…

The description "real-time" varies in meaning depending on the application and acceptable average and maximum latency.

The un-informed usually thinks "real-time" means "hard real-time", but that's seldom necessary, so one can save on additional expense and effort by using a regular linux distro and removing most of the daemons and file location indexing.

I've done real-time development (for Space Shuttle, rocket and balloon projects), and largely all I care about is if a circular buffer can be emptied in time before it fills. That's one technique for avoiding latency variation issues.

The versions of linux that you would normally encounter aim for music real-time, which is about 10 ms latency. 30 ms is considered to be bad.

Most of the pro Yamaha synths use linux as the embedded OS, and some of the code is downloadable (they attempt to comply with the letter of the GPL.) So you can go down to Guitar Center and do a real-time test anytime yourself. :)

The iPhone is pretty good for music, because it was designed to have low latency when playing.

https://www.synthtopia.com/content/2018/02/17/10-years-later-android-audio-quality-still-lags/

https://superpowered.com/androidaudiopathlatency

Looks like Wind River discontinued RTLinux, which was hard real-time (a real shame actually, as it removes one of the few hard real-time options):

https://en.wikipedia.org/wiki/RTLinux

Sadly, most audio APIs are not "half-decent" at zero-latency inter-app routing and minimal mixing/output latency (for a particularly bad example, see https://superpowered.com/androidaudiopathlatency). JACK on Linux supported this but few apps supported it, ALSA and PulseAudio loopback devices don't have it to my knowledge, and PipeWire is the first chance at bringing mainstream non-musician apps to this promised land (but only JACK and PipeWire apps, and Pulse and ALSA apps still have added latency).

Additionally, conventional hardware input/output requires at least two periods of buffering for a stable duplex session (mic input and speaker playback simultaneously). On startup, you fill the output buffer with 2 periods of silence. Every period (when input buffer grows and output buffer shrinks by 1 period), you take 1 period of input audio, feed it through the audio graph (which can take up to 1 period otherwise your speakers run out of audio to play and stutter), then write 1 period of output audio to the speakers. Smaller periods (lower sample/frame count, lower duration in ms) reduce latency, but increase the risk of stutter if your system can't schedule and run all audio apps in 1 period of time.

Android Web Audio latency is over 200-300ms on many devices and is deliberately introduced to kill WebGL browser games that could otherwise bypass the Play Store and its 30% tax. Google has plenty of engineers smart of enough to fix such a glaring latency issue that so many of us have been complaining about for so long but Play Store game profits are in the billions and Google isn't going to let a frictionless browser game experience kill that.

The Superpowered Latency Database[1] has many measurements showing just how bad Android latency is. (A select few devices do OK.) Note that Apple latency measurements look pretty good, because Apple uses different strategies to kill WebGL browser games: (1) Make WebGL slow and lacking features; (2) Prevent browser apps from going fullscreen; (3) Don't even let browser games ask if they can use gyro/tilt control, just block this outright because "Privacy". After all, if Epic Games could just put Fortnite inside mobile Safari their case against Apple wouldn't be so desperate. There's a demo version of Minecraft for the browser that I've never seen work on mobile browsers[2].

[1] https://superpowered.com/latency

[2] https://classic.minecraft.net/

There’s def web clients already out there. I’ve seen satchel, just haven’t come across an actively developed one.

Jssip is a neat library but not a client: https://jssip.net/

Twilio also has a web client.

Taking a cursory search of WebAssembly for voip shows some activity:

Webrtc is a little old and using webassembly to introduce other codecs isn’t entirely new or novel: https://cloudtweaks.com/2020/08/deep-customization-webrtc/

Blazer allows using webassembly to make calls from twilio: https://www.twilio.com/blog/making-phone-calls-from-blazor-webassembly-with-twilio-voice

Another neat use for webassembly and audio is audio production say for podcasts with multiple callers that makes editing a bit easier. https://superpowered.com/js-wasm-overview

Noise cancellation and other dsp effects are something webassembly can meaningfully help with. https://news.ycombinator.com/item?id=30568164