GENIVI and AF_BUS – Here We Go Again?

December 22, 2012

Some interesting links relevant to my current sleeve:

The GENIVI fanbois are seriously committed to D-Bus. As in, like, “GENIVI and D-Bus, sittin’ in a tree…” kind of committed. And they’re not the only ones. D-Bus is used in a surprising number of embedded systems.

So, what do you do when you find out that D-Bus signal delivery causes horrible latency and priority inversion issues when there are multiple subscribers to a signal? GENIVI have decided to try to make D-Bus suck less for the (ab)uses(?) they have in mind, and they’ve put together a fairly targeted patchset toward that end.

It’s hard to argue against a small(-ish) kernel mod that improves performance for one segment of users without affecting others. But looking at the GENIVI AudioManager code makes me skeptical of the ‘problem’ this patchset is intended to solve. Here’s a diagram from the doxygen docs:

AudioManagement

It looks like the idea is for the application logic (policy) bits to live in plugins owned by this daemon, which communicates with various ‘agent’ processes (sources and sinks) over D-Bus . If a navigation app wants to play an audio prompt, it sends a request to the Audio Manager daemon, which will decide whether it’s okay to override sound from the radio, telephone, media player, etc. and perform the appropriate fade-in/fade-out before and afterwards. If you’ve chosen to structure your audio system this way, no wonder D-Bus seems ‘too slow’. The question is: can it ever be fast enough?

I understand why this architecture is attractive to an automotive consortium. They want to create an ecosystem where third party vendors can plug in their stuff with minimal coupling to the rest of the platform (“Android for cars”, in essence.). And it’s hard to get less coupling than a D-Bus-based interface gives you. Vendors don’t have to link with any convenience library, you just tell them: “Here’s the (D-Bus) name of the service you need to connect to, and here are the messages you need to send.”

The problem is that D-Bus was never intended to act as a low-latency pub/sub mechanism. Audio is notoriously susceptible to popping/clicking artifacts due to jitter, which is the reason JACK exists. I’ve seen round-trip latencies for D-Bus requests (as measured by Bustle) measuring several hundreds of milliseconds, so even a 2X improvement will likely prove insufficient. It’s just really hard for me to believe that D-Bus is the right hammer for this particular nail, so I’m not at all surprised by the generally frosty reception the AF_BUS patch received on the kernel mailing list.

Fortunately, D-Bus isn’t a requirement for the GENIVI Audio Manager, it just happens to be used in the ‘reference implementation’. Alternate transports (like ZeroMQ, as suggested in this comment), are permissible. But that brings me to my next nit. Have a look at the signals defined in the Audio Manager interface:

<signal name="NumberOfMainConnectionsChanged"/>

<signal name="SinkAdded">
    <arg type="(qs(nn)nnq)" name="newSink" direction="out"/>
</signal>

<signal name="SinkRemoved">
    <arg type="q" name="removedSinkID" direction="out"/>
</signal>

<signal name="SourceAdded">
    <arg type="(qs(nn)q)" name="newSource" direction="out"/>
</signal>

<signal name="SourceRemoved">
    <arg type="q" name="removedSourceID" direction="out"/>
</signal>

<signal name="NumberOfSinkClassesChanged"/>
<signal name="NumberOfSourceClassesChanged"/>

<signal name="MainConnectionStateChanged">
    <arg type="q" name="connectionID" direction="out"/>
    <arg type="n" name="connectionState" direction="out"/>
</signal>

<signal name="MainSinkSoundPropertyChanged">
    <arg type="q" name="sinkID" direction="out"/>
    <arg type="(nn)" name="SoundProperty" direction="out"/>
</signal>

<signal name="MainSourceSoundPropertyChanged">
    <arg type="q" name="sourceID" direction="out"/>
    <arg type="(nn)" name="SoundProperty" direction="out"/>
</signal>

<signal name="SinkAvailabilityChanged">
    <arg type="q" name="sinkID" direction="out"/>
    <arg type="(nn)" name="availability" direction="out"/>
</signal>

<signal name="SourceAvailabilityChanged">
    <arg type="q" name="sourceID" direction="out"/>
    <arg type="(nn)" name="availability" direction="out"/>
</signal>

<signal name="VolumeChanged">
    <arg type="q" name="sinkID" direction="out"/>
    <arg type="n" name="volume" direction="out"/>
</signal>

<signal name="SinkMuteStateChanged">
    <arg type="q" name="sinkID" direction="out"/>
    <arg type="n" name="muteState" direction="out"/>
</signal>

<signal name="SystemPropertyChanged">
    <arg type="(nn)" name="SystemProperty" direction="out"/>
</signal>

<signal name="TimingInformationChanged">
    <arg type="q" name="mainConnection" direction="out"/>
    <arg type="n" name="time" direction="out"/>
</signal>

This interface is way too ‘fat’ for my taste. It’s trying to predict the needs of every audio system—a doomed exercise. It’s heading down a path with no end in sight, ever. There shouldn’t be so much knowledge encoded in the middleware, IMHO. Stepping back from the problem, it appears that what’s needed is a generic, event-based pub/sub mechanism that supports pre-/post-event hooks. Why limit this to audio-related events? The same framework ought to be able to handle power-moding, persistence, etc. You just need a way for arbitrary processes to emit events, with arbitary subscribers, and arbitary bits of code registered to fire in response. I can’t see any compelling reason to put a straitjacket around the middleware interface.

That just leaves the question of what the messages would look like. I know how I’d do it. People that know me know I love me some Lua. It’s small, fast and there’s an awesome JIT available for it if you need it to be even faster. So, I would make a small request header that looks something like this:

|-- <uint32> --|---- <uint8> ----|---   <string>     ---|
|--PayloadLen--|---- Opcode  ----|--- Request string ---|
|--- 4 bytes---|--1 byte ('Q') --|-PayloadLen - 5 bytes-|

‘Q’ is a mnemonic for ‘Question’ or ‘Query’. The ‘request string’ would be a free-form Lua snippet to be interpreted by the recipient. Responses would look like:

 

|-- <uint32> --|---- <uint8> ----|--  <uint32> --|---   <string>     ---|
|--PayloadLen--|---- Opcode  ----|--Status code--|--- Result string  ---|
|--- 4 bytes---|--1 byte ('A') --|--- 4 bytes ---|-PayloadLen - 5 bytes-|

‘A’ is a mnemonic for ‘Acknowledgement’ or ‘Answer’. The status code would be zero for success, and non-zero for failure. ‘Result string’ would, in the common case, be empty, but could contain arbitrary data (e.g, a Lua table), or an error message if status code is non-zero.

The above Query/ACK messages would be used for (among other things) subscription messages:

Q: "subscribe('source.*')" -- subscribe to 'source' events, e.g., source.removed, source.added, etc.
A: Okay, consider it done.

It would also handle event emissions:

Q: "emit('source.removed', 'dev_IPOD')
A: Okay, consider it done.

For event notifications, the packet would look like this:

|-- <uint32> --|---- <uint8> ----|---   <string>     ---|
|--PayloadLen--|---- Opcode  ----|---  Event string  ---|
|--- 4 bytes---|--1 byte ('E') --|-PayloadLen - 5 bytes-|

The ‘E’ signifies that this is an event (not an acknowledgement):

E: Hey, something just happened. I was asked to give you this Post-It note.
   I can't make any sense of it, but maybe you can...

To hook events, the ‘Event Manager’ (no longer audio-specific) would store registered hooks in a table and do the moral equivalent of:

if (pre_hooks[func_name]) then
  pre_hooks[func_name](arg)
end

func(arg)

if (post_hooks[func_name]) then
  post_hooks[func_name](arg)
end

My Lua’s a bit rusty, so I’m not sure exactly how this would look, but I’m certain there’s a way to wrap pre-/post-handlers around arbitrary functions defined in other modules (sort of like a Python decorator). Need to hook connect() requests and do some sanity-checking? No problem. The nice thing is that the whole framework becomes completely data-driven. The ‘config file’ for the daemons would just be an executable snippet of Lua.

The above suggestion is certainly imperfect, but it at least addresses the fundamental problem, i.e., the need for a fast and flexible, process-distributed event mechanism. Build that and skilled engineers can work out the rest. We desperately need good, solid tools for wiring together distributed state machines and reasoning about their correctness. Whether using Lua as a ‘protocol language’ makes sense is a point for discussion, but regardless—events are a core concept that should be front-and-center in any solution.

{ 2 comments… read them below or add one }

Glenn November 4, 2013 at 6:43 pm

Amen on the GENIVI audio management approach – you’re spot on. With that said, I work for a company that develops IVI software components for embedded Linux. Most of these are written in Lua and in fact we have a Lua-based Audio Management solution that might interest you. It’s already used by a large Tier #1 for an OEM and works very well in their architecture.

I know you may not have the highest regards for D-Bus, but typically it’s not D-Bus that’s the problem but how and where it’s used. We’ve open-sourced our D-Bus/Lua binding if you’re interested . . . it makes things quite a bit easier to to work with D-Bus from Lua. See (https://github.com/xs-embedded-llc/l2dbus) if you’re interested.

Evade Flow November 4, 2013 at 7:56 pm

@Glenn, that’s very cool, I’m glad to see Lua being used in IVI. It makes a lot of sense, since it’s a scripting engine that’s really light on the ‘engine’ part. Javascript is nice for desktop apps, but it isn’t well-suited to the mobile environment if you believe this guy. And I do, based on experience trying to embed Javascript in a realtime simulation engine. (The lack of incremental GC made it unusable. The world would routinely stop for 100+ milliseconds when the garbage collector ran, and nothing we ever tried could fix this.)

Leave a Comment

Previous post:

Next post: