Modal Blog

Thanks for sharing this interesting project and approach!

One suggestion for improvement: Add some more info to your website/GitHub about the need for a provider and which providers are compatible. It took me a bit to figure that out because there was no prominent info about it. Additionally, none of the demos showed a login or authentication part. To me, it seemed like the VMs just came out of nowhere. So at first, I thought "Cloudrouter" was a project/company that gave away free VMs/GPUs (e.g. free tier/trial thing). But that seemed too good to be true. Later, I noticed the e2b.app domain and also then found the note at the bottom of the site that says "Provider selection -> Use E2B provider (default)". Then I got it. However, I should mention that I don't know much about this whole topic. I hadn't heard of E2B or Modal before. Other people might find it more clear.

For those that are wondering about this too, you will need to use a provider like https://e2b.dev/ or https://modal.com/ to use this skill, and you pay them based on usage time.

For most people, before it makes sense to just buy all the hardware yourself, you probably should be renting GPUs by the hour from the various providers serving that need. On Modal, I think should cost about $72/hr to serve Kimi K2 https://modal.com/pricing

Once that's running it can serve the needs of many users/clients simultaneously. It'd be too expensive and underutilized for almost any individual to use regularly, but it's not unreasonable for them to do it in short intervals just to play around with it. And it might actually be reasonable for a small number of students or coworkers to share a $70/hr deployment for ~40hr/week in a lot of cases; in other cases, that $70/hr expense could be shared across a large number of coworkers or product users if they use it somewhat infrequently.

So maybe you won't host it at home, but it's actually quite feasible to self-host, and is it ever really worth physically hosting anything at home except as a hobby?

It depends on what you mean by replacing.

The integrated developer experience is much better on Vapi, etc.

The goal of the Pipecat project is to provide state of the art building blocks if you want to control every part of the multimodal, realtime agent processing flow and tech stack. There are thousands of companies with Pipecat voice agents deployed at scale in production, including some of the world's largest e-commerce, financial services, and healthtech companies. The Smart Turn model benchmarks better than any of the proprietary turn detection models. Companies like Modal have great info about how to build agents with sub-second voice-to-voice latency.[1] Most of the next-generation video avatar companies are building on Pipecat.[2] NVIDIA built the ACE Controller robot operating system on Pipecat.[3]

[1] https://modal.com/blog/low-latency-voice-bot - [2] https://lemonslice.com/ = [3] https://github.com/NVIDIA/ace-controller/

Very cool!

> Unless you've actually watched tokens stream at those rates, the numbers are hard to internalize. This is the rendering.

I built something similar recently, for the same reason: https://modal.com/llm-almanac/token-timing-simulator.

I like that the output rendering is closer to typical UIs -- syntax highlighting in code mode, tool calls, dim-italic reasoning.

One feature mine has that the author, or anyone else who vibe codes their own version after seeing this, might like to steal is modeling the distribution of output latencies. My implementation is hacky (log-normal roughyl estimated from p50, p90, and p99 values), but still, when you set those to realistic values, it recreates the "jitter" you see in many LLM UIs.

antirez is right that generation tok/s isn't flat as a function of context length, which is a weakness of both simualtors.

I built a web viewer for visualizing Gaussian Splat .ply files generated by Apple's ml-sharp, which converts a single photo into a 3D Gaussian Splat in under a second [1].

Features:
- Upload and view .ply file directly in the browser
- Multiple camera trajectory animations (rotate, swipe, shake, forward)
- Interactive orbit controls (drag to orbit, scroll to zoom, right-drag to pan)
- No installation required – runs entirely client-side

The original ml-sharp can render the same video trajectories but requires a CUDA GPU. This viewer lets you explore the 3D output on any device with a browser.

I also added cloud GPU inference via Modal so you can generate splats without a local GPU (free tier available) [2].

[1] https://github.com/apple/ml-sharp

[2] https://modal.com/

Modal's GPU glossary is a good overview about how GPUs work [0]. Karpathy's LLM overview is a good high level overview on LLMs [1]. 3b1b's video (and subsequent videos) on transformers was excellent at helping me understand the math at a high level [2]. This matrix multiplication optimization worklog helped me understand writing better CUDA (not for beginner intro though) [3].

During this process I also asked ChatGPT a lot of questions.

I'm definitely open to suggestions about "how to learn" with all the new tools we have. I felt this has not been straightforward to figure out.

[0] https://modal.com/gpu-glossary

[1] https://www.youtube.com/watch?v=7xTGNNLPyMI

[2] https://www.youtube.com/watch?v=wjZofJX0v4M

[3] https://siboehm.com/articles/22/CUDA-MMM