Read all timestamps into RAM where possible when reading logs
There are cases where we fail to buffer far enough in the future to
solve the timestamp problem, or where we end up choosing to buffer until
the end of time to solve the timestamp problem. These both result in
log reading failures either through CHECKs or OOMs.
Timestamps are tiny. The existence of timestamp_extractor proves that
we can store the whole timestamp problem in memory relatively easily.
When timestamps are stored in separate files, let's just load them at
the start.
This also adds flags to force this behavior on and off. When a log with
data and timestamps mixed in it is found, and the flag forces it on, we
will read the data files twice to extract the timestamps the first time.
I'd like to add more tests to logfile_utils_test to test this all
explicitly, but the multinode_logger tests appear to do a really good
job already.
Change-Id: I38e23836afa980e3e3a839125e78e132066e2c90
Signed-off-by: James Kuszmaul <james.kuszmaul@bluerivertech.com>
diff --git a/aos/events/logging/timestamp_extractor.cc b/aos/events/logging/timestamp_extractor.cc
index 0b22f93..ca7f231 100644
--- a/aos/events/logging/timestamp_extractor.cc
+++ b/aos/events/logging/timestamp_extractor.cc
@@ -19,7 +19,7 @@
int Main(int argc, char **argv) {
const LogFilesContainer log_files(SortParts(FindLogs(argc, argv)));
- const Configuration *config = log_files.config();
+ const Configuration *config = log_files.config().get();
CHECK(configuration::MultiNode(config))
<< ": Timestamps only make sense in a multi-node world.";
@@ -33,8 +33,8 @@
// parts to not be from the same boot.
if (!log_files.ContainsPartsForNode(node_name)) {
// Filter the parts relevant to each node when building the mapper.
- mappers.emplace_back(
- std::make_unique<TimestampMapper>(node_name, log_files));
+ mappers.emplace_back(std::make_unique<TimestampMapper>(
+ node_name, log_files, TimestampQueueStrategy::kQueueTogether));
} else {
mappers.emplace_back(nullptr);
}