April 1st is the day when websites lie to you or create elaborate hoaxes. We’ve generally avoided the former, but we’ve done a few formers, but we also like to use April Fools’ Day as an opportunity to change things up.
So today we’re going to do something different. We’ll talk about my day job. Specifically, we’ll talk about a tool I use in my day-to-day work: cFS.
cFS is NASA’s architecture for designing spaceflight applications. It is open source and designed to be accessible. AND many missions that NASA launches use cFS, which gives it an excellent proven track record. And it was designed and built by people much smarter than me. Which isn’t to say there aren’t WTFs.
The big picture
cFS is a C framework for spaceflight, designed to run on real-time operating systems, although it can fully run on Linux (with or without a real-time kernel) and even Windows. It has three core modules – the Core Flight Executive (cFE) (which provides services around task management and communication between tasks), the OS Abstraction Layer (helps make your code portable across multiple operating systems) and the Platform Support Package (low-level support for hardware connected to the board). Its core concept is that you build “apps”, and the whole thing has an emphasis on the app store. We’ll come back to that. What exactly is an application in cFS?
Well, at their core, “apps” are just actors. They are a block of code with its own internal state, which communicates with other modules via message passing, but basically runs as its own thread (or real-time task, or whatever abstraction is appropriate for your OS).
These applications are connected by a cFS feature called “Core Flight Executive Software Bus” (cFE Software Bus or just Software Bus), which handles subscription management and routing. Under the hood, this exploits the message queue abstraction at the OS level. Since each “application” has its own internal memory and only reacts to messages (or emits messages that others can react to), we avoid almost all the major pitfalls of parallelism.
All of this feeds into the principle of single responsibility, giving each “app” one job. And while we’re throwing around mailers, it also allows us encapsulation (each “app” has its own memory space, which isn’t shared) and helps us design around interfaces – “apps” emit and receive certain messages, which defines their interface. It’s almost like fully object oriented programming in Cu or something like the way BeamVM languages (Erlang, Elixir) work.
Another advantage of this is that we can have reusable applications that provide common functionality that each mission needs. For example, an application DS (Data Storage) records all messages that cross the software bus. LC (Limit Check) allows you to configure expected ranges for telemetry (such as, for example, the temperature you expect the sensor to report) and raise an alert if it falls out of range. there is a SCH (Scheduler) that sends commands to apps to wake them up so they can do useful work (also makes it easy to put apps to sleep indefinitely and minimizes power consumption).
Overall, cFS represents a robust, well-tested framework for designing spaceflight applications.
Even NASA annoys me
However, this is TDWTF, so none of this is all sunshine and roses. cFS isn’t the prettiest framework, and the developer experience can ah… leave a lot to be desired. It’s improving all the time, which is good, but it still has its pain points.
Speaking of continuous improvement, let’s talk versions. cFS is the core flight software framework that hosts your applications (via cFE), and cFS gets new versions. The apps themselves also get new versions. People who write apps and people who write cFS don’t always coordinate on this, which means that when cFS adds a significant change to its API, you have to play “which version of cFS and App X play well together”. And since everyone has different release tagging practices, you often have to go through the commits to find the last version of an app that was compatible with your version of cFS and see things like releases marked “compatible with Draco rc2 (mostly)”. The “download apps from the App Store and they just work” goal is definitely not really happening.
Or, this, from the current cFS readme:
List of compatible cFS applications
The following applications have been tested against this release:
TBD
Messages in cFS represent structs. Which means that when applications want to send messages to each other, they need the same structure definitions. This is just a pain – to agree on which application should own which message, who needs the definition and how when we transfer the definition to them it’s just a big mess. It’s such a big mess that newer versions of cFS have switched to using “Electronic Data Sheets” – XML files that describe structures, which doesn’t really solve the problem, but adds XML to the mix. EDS at least makes it easy to share definitions with non-C applications (popular terrestrial software is written in Python or Java).
Messages must also have a unique “message ID”, but the MID is not just an arbitrary unique number. It secretly encodes important information, such as whether this message is a command (an instruction to take an action) or telemetry (data to be output), and if you choose a bad MID, everything breaks. Also, keeping MID definitions unique across many different applications that don’t know other applications exist is huge an issue. A general solution that people use is to attach some sort of CSV file and a code generator that handles it.
These MIDs also do not exist outside of cFS – they are an abstraction unique to cFS. cFS converts them behind the scenes into different parts of the “space packet header”, which is the primary packet format for the SpaceWire network protocol. This means that in realistic implementations where your cFS module needs to talk to components that don’t run cFS – your MID also represents key header fields for the SpaceWire network. It’s incredibly overloaded, and the conversions are hidden behind C macros that you can’t easily debug.
But my biggest gripe is the crafting tool. Everyone at work knows they can send me climbing walls just by whispering “cFS builds” in my ear. It’s a nightmare (it’s gotten better in newer versions, I believe, but because of the whole “no sync between app versions and cFE” issue, we’re not using the new version). It starts with makewho calls CMakewhich also calls makebut also calls CMake again in a way that does not allow variables to propagate to other layers. cFS doesn’t provide any targets that you connect to, but instead requires that any applications you want to use be directly injected into the cFS source tree, making it incredibly difficult to build only parts of cFS for unit testing.
Oh, speaking of unit testing – cFS offers mocking of all its internal functions; mocks that always return an error code. This is intended to encourage developers to test their own failure paths in the code, but I also like to test ours the path to success also.
Abstract
Any tool that you use regularly will be one that you are intimately familiar with; the good parts often fade into the background, and the pain points are the things you notice day in and day out. I definitely feel that way after two years of working with cFS.
I think that, at its core, the programming concepts it brings to work are low-level, embedded Cu good. That’s certainly better than having to write this architecture myself. And for all its warts, it was designed and developed by people who are extremely security conscious and expect you to be too. It’s been used in many missions, from hobbyist cube satellites to Mars rovers, and that proven track record gives you a good degree of confidence that your mission will be safe if you use it.
And since it’s Open Source, you can try it yourself. The cFS-101 tutorial gives you a good starting point, complete with a downloadable VM that walks you through building a cFS application and communicating with it from simulated ground software. It’s a very different way to approach C programming (and makes it easier to align with C standards, like MISRA), and frankly, an actor-oriented mindset is a good attitude to have a lots of different architectural problems.
Peregrine
If you’ve been following space news at all, you may already know that our Peregrine lander is broken. I can’t say anything about it until the official audit releases its findings, but all indications are that it was largely a hardware problem involving valves and high pressure tanks. But I can tell that most of the avionics on it were connected to some sort of cFS instance (there were several cFS nodes).
[Advertisement]
Continuously monitor your servers for configuration changes and report when configuration drift occurs. Get started with Otter today!