Avionics Ethernet Hardening

Summary

This RFD proposes the next hardening direction for the Ethernet work in the Arty FPGA projects.

The current prototype has demonstrated an incremental path from PHY bring-up through UDP/CCSDS command and telemetry exchange. Its newest checkpoint replaces single-entry clock-domain mailboxes with bounded queues, preserves ordering, and makes overload visible. That is useful engineering evidence, but it is not a flight design.

The proposed direction is to turn the endpoint into a small, deterministic avionics service with:

• protocol termination at each trust boundary;
• fixed, versioned command and telemetry contracts;
• bounded queues with reserved capacity and explicit overload behavior;
• end-to-end command identity, freshness, integrity, and outcome reporting;
• fault containment per Ethernet port;
• source timestamps and time-tagged execution where synchronization matters;
• local watchdogs, limits, and safe-state behavior at every remote endpoint;
• a verification ladder that proves each property before adding network scale or flight-hardware complexity.

The immediate recommendation is not to build a transparent Ethernet switch or claim spacecraft qualification. The next prototype should add one explicit remote-port boundary to the Project 13 queue-hardening work and prove that one command can be accepted, forwarded, executed, completed, or failed without ambiguity.

This document is in discussion. I am looking for peer review on the system boundary, failure semantics, timing assumptions, and the smallest useful next hardware experiment.

Why this needs discussion

Ethernet makes the physical connection familiar, but it does not make an avionics network deterministic, safe, or fault tolerant by itself. The difficult questions live above and around the wire:

• What happens when a queue fills?
• Can stale, duplicated, malformed, or replayed commands cause action?
• How does the sender learn whether a command was merely received or actually completed?
• Can one failed or babbling endpoint consume another endpoint's resources?
• What survives a partial reset or a link interruption?
• Where is time measured, and how old may data be when it is used?
• Which safety properties must remain local even when the network or central controller fails?

The current repository deliberately answers these questions one checkpoint at a time. The Ethernet board bring-up ladder records the physical evidence already gathered. The timing and spacecraft synchronization note captures the timing, topology, standards, and fault-containment research behind this proposal.

Current evidence

The Arty work has demonstrated:

1. deterministic PHY clock and reset behavior;
2. raw Ethernet command reception and status transmission;
3. IPv4/UDP commands carrying CCSDS-style packets;
4. command/response telemetry observable from a host;
5. named host-side actions and repeatable bench evidence;
6. a fixed 64-bit command adapter with state telemetry;
7. four-entry asynchronous RX and TX queues with ordered delivery, sticky overflow indication, saturated overflow count, and deterministic reset.

Project 13: Ethernet Command Queue Hardening is the direct baseline for this RFD. It asks whether a bounded burst can cross Ethernet clock domains without silent loss or reordering.

Project 13 intentionally does not yet provide:

• complete Ethernet FCS, IPv4, and UDP validation;
• duplicate suppression or replay protection;
• deadlines or command freshness;
• PTP/gPTP synchronization or scheduled traffic;
• per-port fault containment across multiple remote endpoints;
• EDAC, TMR, scrubbing, redundant links, or watchdog recovery;
• radiation, environmental, EMC, safety, or flight qualification.

Those omissions are not hidden defects in the checkpoint. They define the work still needed.

Proposed system boundary

The baseline architecture is a protocol-terminating Ethernet communications module (ECM) between the onboard computer (OBC) and independently connected remote FPGA endpoints.

The ECM is a gateway, not a transparent Layer-2 bridge:

• it validates and terminates the OBC-side protocol;
• it routes only known message types to statically configured device IDs;
• it creates a separate remote-side transaction;
• it owns bounded resources per port;
• it reports stable outcomes and reason codes;
• it prevents a remote endpoint from injecting arbitrary traffic into the OBC network or another remote link.

This boundary is intended to make authority, resource ownership, and failure containment reviewable. It does not decide whether the OBC and ECM eventually reside on separate boards or in the processor and programmable logic of one SoC.

Command lifecycle

The command contract should distinguish these states:

1. received — bytes arrived at an interface;
2. validated — framing, length, version, integrity, destination, and authorization checks passed;
3. accepted — the receiver reserved the resources required to continue;
4. executed — the remote endpoint started the requested operation;
5. completed — hardware-observed completion met the command criteria;
6. failed — the command reached a terminal failure with a stable reason.

An acknowledgement at one state must not be mistaken for a later state. Every request and outcome should carry enough identity to correlate across a reset, retry, or delayed response:

• protocol version;
• source and destination device IDs;
• boot or session identity;
• transaction ID;
• sequence number;
• command type and parameters;
• creation time and deadline where applicable;
• end-to-end integrity value;
• outcome state and reason code.

Retries must be command-specific. Repeating a state query may be harmless; blindly repeating an actuator pulse may not be. The protocol therefore needs an idempotency rule rather than one global retry policy.

Hardening requirements

These are prototype requirements for learning and verification. Flight requirements must come from mission timing, reliability, safety, radiation, thermal, EMC, and qualification constraints.

Fault model for the next prototype

The next increment should cover faults that can be injected and observed on the bench:

• corrupt or truncated frame;
• unsupported protocol version or command;
• duplicate or stale transaction;
• RX or TX queue exhaustion;
• remote endpoint that sends continuously;
• link loss before acceptance, after acceptance, and during execution;
• OBC-side, ECM-side, and remote-side reset at each command lifecycle state;
• missing or delayed terminal outcome;
• clock-domain reset skew;
• counter saturation and telemetry loss.

It should not pretend to close radiation-induced upset, latent hardware fault, common-cause power or clock failure, Byzantine behavior, or independent safety inhibit allocation. Those require hardware choices and system-level analyses that do not yet exist.

Timing position

Ordinary Ethernet provides neither deterministic end-to-end latency nor synchronized time. On a direct full-duplex link, propagation is usually small; PHY pipelines, buffering, frame serialization, switching, software, and contention dominate the useful timing budget.

For the prototype:

• use direct full-duplex links;
• fix link rate and maximum frame size;
• disable Energy Efficient Ethernet;
• use bounded queues and traffic rates;
• timestamp as close to the wire or data source as practical;
• measure the complete path on representative hardware;
• keep actuation local unless a synchronized, time-tagged contract is required.

PTP or IEEE 802.1AS may later distribute time, but neither should be adopted without an error budget and timestamp architecture. Time-Triggered Ethernet or TSN may later bound contention, but scheduling machinery is premature until the traffic inventory and command lifecycle are stable.

Alternatives considered

Transparent Layer-2 switching

This is familiar and flexible, but it broadens the fault and trust domain. It also makes arbitrary forwarding, shared buffering, broadcast behavior, and address learning part of the safety and verification problem. It is rejected for the prototype baseline.

One shared switched network

A managed TSN or TTEthernet network may be the correct flight architecture when scale, redundancy, or deterministic scheduling demands it. It is not the smallest experiment that closes the current uncertainty around lifecycle and fault containment.

Raw Ethernet only

Raw frames keep the FPGA small and can be deterministic on one link. Static UDP/IPv4 is preferred for the next discrete-board prototype because existing host tools and CCSDS-style payloads already work across that boundary. The higher-level lifecycle must remain independent of this transport choice.

Same-SoC shared memory

If the OBC software and ECM logic share a SoC, shared-memory command and outcome rings may be better than an internal Ethernet link. This changes the transport, not the message identity, queue ownership, reset generation, or lifecycle requirements.

Add redundancy immediately

Redundancy before command semantics are stable can duplicate ambiguity and introduce split-brain authority. A single-channel design should first produce unambiguous outcomes and bounded fault behavior. Mission reliability and hazard analysis should then determine whether dual ECMs, cross-strapping, or replicated links are required.

Verification ladder

The proposal extends the existing bring-up ladder rather than replacing it.

The recommended next build is Step 1 or Step 2. A full multiport switch would skip the most important unresolved contract.

Proposed determination

For the research baseline:

1. Use a protocol-terminating ECM gateway with one independently bounded point-to-point endpoint per remote FPGA.
2. Keep flight applications behind one message-based hardware service so they do not depend on the transport backend.
3. Use versioned CCSDS Space Packets over static UDP/IPv4 for the next discrete-board prototype.
4. Require explicit command identity, freshness, integrity, capacity reservation, lifecycle outcomes, and reason codes.
5. Keep per-port queues, policing, counters, reset state, and recovery behavior bounded and independently observable.
6. Timestamp sensor data at its source and use time-tagged execution when synchronized action is required.
7. Keep safety limits, watchdogs, and safe-state behavior at the remote endpoint.
8. Treat the physical layer, redundancy, hazard controls, and flight qualification as open system decisions.

Questions for peer review

I would especially value a peer's response to these questions:

1. Is the ECM boundary in the right place, or does it hide a responsibility that should remain in the OBC or remote endpoint?
2. Are the six lifecycle states sufficient and unambiguous? Which outcomes would you combine or split?
3. What is the most dangerous reset or retry case missing from the proposed fault model?
4. Is static UDP/IPv4 a reasonable next prototype transport, or would raw Ethernet reveal the same architectural risks with less logic?
5. Which observability fields would you require before trusting a board test?
6. What minimum traffic inventory and timing budget should exist before discussing TSN, PTP/gPTP, or TTEthernet?
7. Which safety functions must be physically or logically independent of the ECM for an actuator use case?
8. Is the next experiment small enough to falsify the architecture, or is it already trying to prove too much?

Open system questions

Before selecting flight hardware or freezing an interface control document, we still need:

• the physical partition between OBC, ECM, and remote endpoints;
• the number and role of remote FPGAs;
• worst-case rate, frame size, and burst per link;
• maximum latency, jitter, data age, execution skew, and recovery time;
• the hazardous-command inventory and allocated inhibits;
• required service after any single failure;
• the available clock, PPS, or time-distribution source;
• environmental, radiation, EMC, connector, cable, and qualification constraints;
• the required behavior when a terminal outcome is lost after the action has already occurred.

The last question is central. A communications design cannot safely define retries without knowing whether the action is idempotent.

References

• Arty Ethernet board bring-up ladder
• Arty Ethernet timing and spacecraft synchronization research
• Project 13 queue-hardening checkpoint
• NASA core Flight Executive Application Developer's Guide
• CCSDS 133.0-B-2: Space Packet Protocol
• RFC 768: User Datagram Protocol
• IEEE/SAE 802.1DP-2025: TSN Profile for Aerospace Onboard Ethernet Communications
• IEEE 802.1AS-2025: Timing and Synchronization for Time-Sensitive Applications
• SAE AS6802A: Time-Triggered Ethernet
• ECSS-E-ST-50-16C: Space Engineering — Time-Triggered Ethernet
• NASA Software Safety Analysis guidance
• RFC 3819: Advice for Internet Subnetwork Designers

Discussion status

This RFD proposes a research and verification direction. It is not a flight design approval, a safety determination, or a hardware selection.

The decision should remain in discussion until a peer has reviewed the system boundary and command lifecycle and the first two verification steps have an agreed contract.