Methodological Pitfalls in Predicting Ship Turnaround Time at Brazilian Ports: An Empirical Audit and a Reproducible Pipeline

summary

The gist

Two recent studies on ship turnaround time at Brazilian ports report models with R-squared scores above 0.80 on data from the national waterway transportation agency.

In short

Studies showing good accuracy for ship turnaround time models at Brazilian ports were found to be misleading due to structural data leakage. The audit revealed that 97% of test events were in the training set, inflating results. A new pipeline using berthing-level aggregation, conditional imputation, and cargo embeddings improves performance on operational targets by capturing target-specific effects.

Key concepts

Structural Data Leakage
This occurs when information from the testing data accidentally appears in the training data. In this paper, it was found that 97% of berthing events in the test set were already present in the training set under a random split, causing models to appear highly accurate when they are not truly predictive.
Berthing-Level Aggregation
Instead of analyzing data at the individual row level, this method groups all cargo attributes for a single berthing event together. This ensures that each data point in the model represents one complete ship stay rather than just one piece of cargo information, which fixes the leakage issue.
Cargo Embeddings
These are numerical representations (vectors) created from complex text descriptions of cargo, weighted by its mass. They allow a model to understand the semantic meaning of cargo—like destination or handling requirements—and use this rich context to make more accurate predictions for specific time targets.
Target-Specific Heterogeneity
This means that the impact of a variable (like cargo) is different depending on which prediction you are trying to make. The paper shows that cargo embeddings significantly improve predictions for operational times (like loading), but have little effect on administrative times, highlighting this crucial difference.

Terminology used across episodes

This episode discusses

The paper

Methodological Pitfalls in Predicting Ship Turnaround Time at Brazilian Ports: An Empirical Audit and a Reproducible Pipeline · Read on arXiv

Eduardo Caruso Barbosa Pacheco, Reynaldo Pereira Martins, Alexandre dos Santos Gualberto, Joao Vitor Souza ğ Germano, Milton Miranda Neto, Francisco Louzada

Institute of Mathematical and Computer Sciences, University of Sao Paulo

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Methodological Pitfalls in Predicting Ship Turnaround Time at Brazilian Ports".

Jane: Here's the gist of the paper. Two recent studies on ship turnaround time at Brazilian ports report models with R-squared scores above 0.80 on data from the national waterway transportation agency.

Tom: We audit their published datasets and reveal these results are inflated by structural data leakage.

Jane: Specifically, 97 percent of berthing events in their random row-level split appear in both training and testing partitions.

Tom: We demonstrate that a model’s R-squared drops from 0.82 to 0.20 when the data is properly split at the berthing level.

Title and authors: Tom: So we're looking at this paper called "Methodological Pitfalls in Predicting Ship Turnaround Time at Brazilian Ports: An Empirical Audit and a Reproducible Pipeline." It sounds pretty technical, so Jane, can you tell us what the main point is without getting bogged down in the details?

Jane: Well, basically, they are taking two existing studies that claimed they had really good models with R squared scores over zero point eight on data from the national waterway transportation agency. But this paper audits those datasets and shows that those high scores are actually inflated because of something called structural data leakage.

Tom: Data leakage? That means the training data accidentally knows too much about the test data, right? Like, if you randomly split your rows, you might end up with the same event in both groups.

Jane: Exactly. The paper found that ninety-seven percent of berthing events that are in their random row-level split actually show up in both the training and testing partitions. That’s a massive amount of leakage.

Lu: This means their reported performance numbers, like those R squared scores above zero point eight, aren't real performance metrics on new data; they're just artifacts of how they split the data <ref:2611.14713#pg1>.

Meng: From a deployment standpoint, if your model is trained on leaked information, it will look great in testing but fail completely when you put it into production with real, unseen data. It’s not scalable or reliable at all <ref:2611.14713#pg2>.

Tom: So, they are showing us how to fix that initial problem by properly splitting the data at the berthing level instead of just randomly splitting every single row.

Jane: Right, and beyond that basic fix, they found other things going wrong with the original models too, like survivorship bias and undocumented target truncation.

Lu: They also pointed out variables in the original models that mathematically reconstruct the target variable <ref:2611.14713#pg3>.

Meng: That’s bad because you end up building a model that is just memorizing the data structure instead of actually learning a useful pattern for predicting time <ref:2611.14713#pg3>.

The paper's summary: Tom: So, what's the core proposal they are making to fix all these issues? What’s the big picture fix described in this paper?

Jane: The main proposal is a reproducible pipeline. They suggest you aggregate everything to the berthing level first. This means one row represents one ship visit to a berth, which fixes that ninety-seven percent leakage we talked about earlier <ref:2611.14713#pg3>.

Lu: They take the raw data and sum up numeric cargo attributes and use weighted one-hot sums or first operations for things that are categorical <ref:2611.14713#pg3>. That’s how they handle the data grain issue.

Meng: And for those high cardinality text variables, like the cargo group, they don't just drop them; they use multilingual sentence embeddings weighted by cargo mass <ref:2611.14713#pg2>.

Tom: Wait, so they are using these embeddings to capture the meaning of the cargo instead of just treating it as a random label? That seems like a smart way to handle that complexity.

Jane: It is. They use conditional imputation too, which means instead of just deleting rows with missing values, they apply rules based on what those missing values mean structurally <ref:2611.14713#pg3>.

Lu: And the results from this new pipeline are pretty interesting; they tested six different time targets and thirty random seeds, and cargo embeddings showed statistically significant improvements on four of those operational targets <ref:2611.14713#pg2>.

Meng: Four out of six operational targets improved significantly, but the administrative waiting times didn't see any change at all. That’s a key finding about where the signal is strongest <ref:2611.14713#pg2>.

Tom: So what does that mean for us when we try to predict turnaround time? It suggests that how you represent the cargo really matters for what part of the process you are predicting.

The paper's improvements: Jane: The paper details how this new pipeline actually improves things, and it’s not just one thing; it’s a whole set of changes <ref:2611.14713#pg2>. They showed that using the semantic cargo representations yields statistically significant gains on four operational targets.

Lu: Specifically, adding the cargo group embedding yielded a statistically robust improvement on TStay, which is the time at stay, with a difference of one point three three hours and a p value less than ten to the power of negative seven <ref:2611.14713#pg2>.

Tom: one point three three hours. That’s tangible data. But are these improvements consistent across all time predictions, or is it just better at one specific thing?

Jane: It isn't uniform; the results show that cargo embeddings are specific to each target metric rather than being the same across the whole stay process <ref:2611.14713#pg2>.

Meng: So, cargo information helps predict things like TStay, wait to moor time, time moored, and operation time <ref:2611.14713#pg2>, but it doesn't help much with the administrative ones like wait start operation or wait unmooring <ref:2611.14713#pg2>.

Lu: That pattern is interesting because it suggests that cargo identity has a very large and statistically robust effect on the operational components of the stay, like loading and waiting to moor <ref:2611.14713#pg2>.

Tom: So, in simpler terms, if you want to cut down physical time at port—like loading or waiting to moor—focus your cargo data efforts there because that's where the model actually sees the biggest benefit.

Conclusion: Jane: So to wrap up, the main message of this paper is that relying on simple random splits and basic feature handling leads to misleading performance numbers when predicting ship turnaround time at Brazilian ports. The authors show you how to build a proper pipeline using aggregation, conditional imputation, and semantic cargo embeddings <ref:2611.14713#pg2>.

Tom: They also highlighted that the combination of the data release and their original methodology choices produced numbers whose interpretation as predictive performance is misleading, so you need this corrected pipeline to see what's actually happening <ref:2611.14713#pg2>.

Lu: The paper supports a holistic management approach where cargo data is useful for physical planning, but managing administrative performance requires a new approach to process management <ref:2611.14713#pg2>.

Meng: I’d add that the authors suggest managers should look at operational indicators that the original model missed, like document processing status and inspection requirements <ref:2611.14713#pg2>.

Lalam: I'm confident in this direction; semantic cargo representations facilitate more informed predictions regarding waiting time, time at berth, handling time, and total turnaround time <ref:2611.14713#pg2>.

Tom: So we’ve seen how structural leakage ruins the numbers and what a fixed pipeline looks like when you properly aggregate cargo to the berthing level <ref:2611.14713#pg3>. That's all for this paper.

Jane: We’ll be back after the break with another look at some of these other interesting papers on arXiv.

More episodes

← Home