Data Science
Signal over noise.
Models, analysis, statistics, ML foundations.
8 notes
Evaluating and observing a RAG system: judges, traces and where time goes
A retrieval system has many moving parts, so 'it feels worse' is not a diagnosis. Two habits fix that: evaluate with a mix of code checks, an LLM-as-judge and human feedback; and trace every request so you can see which step failed and where the latency actually lives (almost always the model, not retrieval).
Reranking and chunking: the two levers that quietly decide retrieval quality
Fast retrieval casts a wide net; a slower cross-encoder reranks the shortlist for precision. And how you cut documents into chunks decides what a single vector even means. Both are unglamorous, and both move retrieval quality more than the model choice does.
How search actually works: keyword, semantic and hybrid
Keyword search matches exact words (BM25). Semantic search matches meaning (embeddings). Each fails where the other shines, so the honest default is hybrid: run both and fuse the rankings. This is the retrieval layer under any system that answers from a document base.
Vector databases and the ANN trade-off
Exact nearest-neighbor search compares your query to every vector, which is perfect but linear and dies at scale. Approximate methods like HNSW navigate a layered graph to find near-perfect matches in logarithmic time. The whole trade is a little accuracy for enormous speed.
Does it actually work? Absolute risk, ARR and NNT
The same result can be sold as a 60% improvement or a 3-point one, depending on whether you report relative or absolute change. Medicine has clean tools for this (absolute risk reduction, number needed to treat) and they are the honest way to judge any intervention, including training methods and supplements.
Survival analysis beyond mortality: time-to-event for rehab and sport
Medicine uses survival analysis to model time until death, but the machinery answers a far more general question: how long until an event happens, when some people have not had it yet? That censoring problem is exactly what breaks naive stats in return-to-play, reinjury and program dropout.
Which wearable metrics to trust (and which to question)
Steps and heart rate are accurate enough for most people. Energy expenditure, sleep stages, stress and 'readiness' scores are much softer, so read them as trends and not as verdicts. Knowing which number deserves a decision is half the job.
N-of-1: each person is their own baseline
Population averages describe a crowd, not a person. N-of-1 analysis looks at repeated measurements inside one individual over time. It's the right frame for questions like 'what happens to my energy when I sleep less than my own average?'