NCC Group Research

I'm not a RISC-V expert, primarily relying on open-source firmware community knowledge and the opinion of such figures as Ron Minnich, but AFAIK, most firmware for production RISC-V deployments is closed-source, so we can't say if vendors use OpenSBI implementation or some modified version for their malicious purposes. If I need to be corrected, please point me to products that state how they transparently use SBI. There is evidence of transparent use of SMM in coreboot. SBI is not the only RISC-V TEE. There are also other concepts, like MultiZone [1] and PMP-based Keystone [2], advertised as trusted execution environments

The openness of specification only matters a little here - UEFI specification about management mode is also open. But IBVs screwed us many times [3] by implementing low-quality SMIs that could elevate privileges, and the spec does not guarantee that implementers follow it. Lack of tooling makes compliance difficult - it slowly changes with various startups seeing that as an opportunity (e.g., Binarly, Eclypsium?).

My point is that SMM as TEE is not a unique x86 feature, and other significant architectures have similar mechanisms. OS has a means of figuring out that it was in SMI, so it is not entirely hidden but still has superpowers. In the CNC world SMI latency is measured from OS [4].

Trusted execution environments are hammers and can be used for a good purpose in a transparent way and for malicious purposes. Further typically depends on the…

Allocator efficiency was the primary concern in the past (both speed and packing), which is why there are several things in malloc which is obviously bad for allocation safety.

The allocators in the past have tried to use the space it is not currently using for user code for its own book-keeping, which is very efficient when it comes to usage of space. So the free-lists are interleaved with the actual allocations to be as cheap as they can be (not in a different page).

Oddly enough, packing all the metadata in a single area might actually be faster though at lower scales might be worse for the data itself.

I've used talloc[1] in the past, which traded some of the efficiencies for safety (I have my own sins trying to copy, but simplify pool based allocators for PHP apc with some tricks[3] borrowed from xcache).

If the author is reading, the missing magic I've always missed with jemalloc are the .gdb macros for walking the allocator and do things like print the allocations by size/content [2]. You can write them in python now and that can let you print them out neatly in a graphviz dot structure with linkages, which is the sort of project which you hand out to an intern who just joined.

[1] - https://talloc.samba.org/talloc/doc/html/libtalloc__dts.html

[2] - https://research.nccgroup.com/2015/08/11/libtalloc-a-gdb-plugin-for-analysing-the-talloc-heap/

[3] - http://notmysock.org/blog/2009/Jan/08/

> Any evidence in this area?

See danShumway's post below. People are regularly posting exploits on twitter, including getting the system to dump it's prompt.

May I ask politely, are you a programmer, and have you secured system's previously? It will change the way I approach trying to carry my message across.

For background, a finished LLM is a blackbox. You can't program the LLM in the box in the traditional sense, because we don't fully understand what happens in the box at a level where we can "code" it.

Judging the security of a filter by the cases where it works is a very bad way to judge security. Blocklists ARE NOT SAFE because it is impossible to account for the infinite variety of things that can be tried.

Here's a whitepaper on the difficulties. There's been lots of writing about this:

https://research.nccgroup.com/wp-content/uploads/2020/07/ncc-group-whitepaper_the-disadvantages-of-a-blacklist-based-approach-to-input-validation.pdf

Now, this has been shown to be difficult for really constrained scenarios, like SQL and so forth, but English has a million words, for starters.

Here are additional resources about specific defense techniques for prompt attacks:

NCC Group:
Exploring Prompt Injection Attacks
https://research.nccgroup.com/2022/12/05/exploring-prompt-injection-attacks/

Preamble:
Ideas for an Intrinsically Safe Prompt-based LLM Architecture
https://www.preamble.com/prompt-injection-a-critical-vulnerability-in-the-gpt-3-transformer-and-how-we-can-begin-to-solve-it

@Riley, hello, I wanted to say hi and I would love to connect with you if you have time, as I also work in the prompt safety space and would be honored to brainstorm with you someday. Would you like to start a message thread on a platform that supports it? I think the research you are doing is amazing and would love to bounce some ideas back & forth. I was the one who discovered some version of prompt injection in May 2022 while researching AGI safety and using LLM as a stand-in for the hypothetical AGI. You could email me at upwardbound@preamble.com to reach me if you would like! Sincerely, another prompt safety researcher

When the input is first selected here's will be a hot plug detect, EDID request and an EDID response. That response can be up to 32kB so anywhere up to ~2 seconds depending on how many extension blocks are in use by the display. Here's a nice breakdown on more of what happens there: https://research.nccgroup.com/wp-content/uploads/2020/07/hdmi_-_hacking_displays_made_interesting.pdf.

When a suitable set of image parameters have been agreed HDCP (may) then step in to ruin your day. Again a nice walkthrough of that is available here: https://blog.cryptographyengineering.com/2012/08/27/reposted-cryptanalysis-of-hdcp-v2/.

As other commenters have correctly noted, there's definite room for improvement. One thing to remember though is this involves hardware that can be in operation for the order of decades so there's a lot of cruft that accretes in order to maintain compatibility. Frankly it's mildly impressive that for the most part all the above happens in silence and continues to (mostly) work.

Do you mean Ethernet over HDMI? Even though there was a standard and special HDMI cables that claimed compatibility, I don’t think any devices actually implemented it. Well perhaps one device according to this security paper[1] which basically means it’s useless.

I control access by connecting a Samsung TV to my Ethernet network but keep it configured to use WiFi most of the time. I’ve never entered a WiFi password so the network is effectively disconnected until I configure the TV to use the wired connection.

The TV prompts me to use the wired connection when I reboot my router and the link drops and comes back up but I can cancel switching to wired at that point.

[1]: What the HEC? Security implications of HDMI Ethernet Channel and other related protocols
Andy Davis, Research Director NCC Group
https://research.nccgroup.com/wp-content/uploads/2020/07/44con_hdmi_ethernet_channel_andy_davis_ncc_group_slides.pdf