Track matching forwarded times for log file sorting
Previously, the only thing we knew was a loose ordering based on part
index. That isn't very conclusive. This now provides us with
corresponding timestamps for when things cross the network boundary in a
way that we think should tell us definitively what came first.
There are 2 main problem cases. Let's say we have 2 channels. /a which
is reliable, and /b which isn't, both sent from the same remote node.
case 1: /a -> boot 0 received on boot 0.
/b -> boot 1 received on boot 0.
We start logging after both messages arrive.
case 2: /a -> boot 0 received on boot 0.
/b -> boot 0 received on boot 0.
We log for a bit, then reboot. More messages show up when we reconnect.
/a -> boot 0 received on boot 1.
/b -> boot 0 received on boot 1.
In case 1: we only have a reliable timestamp from boot 0, but that
reliable timestamp makes it clear that /a was before /b, so boot 0 was
before boot 1.
In case 2: we have the same reliable timestamp, so that tells us nothing.
The unreliable timestamps though tell a different story. /b will be after
/a, since any messages on /b generated before the reboot won't get
delivered. So, we get an ordering constraint saying that any sent /b's
on the second boot were after /b on the first boot.
We believe that any other cases are covered by the same mechanism.
Without fully implementing and debugging the sorting code, we won't know
for certain. Sanjay and I have been unable to break the logic so far.
Change-Id: I990bf249b18bf43072997cdb099ac66c2fa8fc57
Signed-off-by: Austin Schuh <austin.schuh@bluerivertech.com>
diff --git a/aos/events/logging/log_namer.h b/aos/events/logging/log_namer.h
index d9f42db..8a44c97 100644
--- a/aos/events/logging/log_namer.h
+++ b/aos/events/logging/log_namer.h
@@ -44,14 +44,10 @@
// Rotates the log file, delaying writing the new header until data arrives.
void Rotate();
- // TODO(austin): Copy header and add all UUIDs and such when available
- // whenever data is written.
- //
- // TODO(austin): Add known timestamps for each node every time we cycle a log
- // for sorting.
-
- void UpdateRemote(size_t remote_node_index,
- const UUID &remote_node_boot_uuid);
+ void UpdateRemote(size_t remote_node_index, const UUID &remote_node_boot_uuid,
+ monotonic_clock::time_point monotonic_remote_time,
+ monotonic_clock::time_point monotonic_event_time,
+ bool reliable);
// Queues up a message with the provided boot UUID.
void QueueMessage(flatbuffers::FlatBufferBuilder *fbb,
const UUID &node_boot_uuid,
@@ -69,6 +65,28 @@
size_t parts_index() const { return parts_index_; }
const Node *node() const { return node_; }
+ // Datastructure used to capture all the information about a remote node.
+ struct State {
+ // Boot UUID of the node.
+ UUID boot_uuid = UUID::Zero();
+ // Timestamp on the remote monotonic clock of the oldest message sent to
+ // node_index_.
+ monotonic_clock::time_point oldest_remote_monotonic_timestamp =
+ monotonic_clock::max_time;
+ // Timestamp on the local monotonic clock of the message in
+ // oldest_remote_monotonic_timestamp.
+ monotonic_clock::time_point oldest_local_monotonic_timestamp =
+ monotonic_clock::max_time;
+ // Timestamp on the remote monotonic clock of the oldest message sent to
+ // node_index_, excluding messages forwarded with time_to_live() == 0.
+ monotonic_clock::time_point oldest_remote_unreliable_monotonic_timestamp =
+ monotonic_clock::max_time;
+ // Timestamp on the local monotonic clock of the message in
+ // oldest_local_unreliable_monotonic_timestamp.
+ monotonic_clock::time_point oldest_local_unreliable_monotonic_timestamp =
+ monotonic_clock::max_time;
+ };
+
private:
// Signals that a node has rebooted.
void Reboot();
@@ -88,7 +106,7 @@
std::function<void(NewDataWriter *)> close_;
bool header_written_ = false;
- std::vector<UUID> boot_uuids_;
+ std::vector<State> state_;
};
// Interface describing how to name, track, and add headers to log file parts.
@@ -185,7 +203,7 @@
// Creates a new header by copying fields out of the template and combining
// them with the arguments provided.
aos::SizePrefixedFlatbufferDetachedBuffer<LogFileHeader> MakeHeader(
- size_t node_index, const std::vector<UUID> &boot_uuids,
+ size_t node_index, const std::vector<NewDataWriter::State> &state,
const UUID &parts_uuid, int parts_index) const;
EventLoop *event_loop_;