Tools That Last in Data Science
After 10 years in data science, the tools that stayed were rarely the flashiest ones. They are the ones that keep solving the same boring, recurring work well.
TL;DR: The tools that survive solve recurring problems with low friction. After 10 years in data science, the tools that stayed were rarely the flashiest ones — they were the ones that keep paying rent. SQL still gives the first useful answer because it tests an idea close to the data, before a notebook, feature pipeline, dashboard, or model exists. LightGBM and XGBoost remain serious baselines for structured tabular problems because they are efficient, mature, and hard to beat without a clear reason for more complexity. Pydantic holds its place by making assumptions explicit at system boundaries, so API inputs, configuration, pipeline handoffs, and structured outputs are easier to validate, review, and maintain. What did not stick were tools that required significant setup without enough repeated value, or that solved a problem rarely encountered. The longer you work, the less you optimise for novelty and the more you optimise for tools you still reach for when there is real work to do. The most durable tools are rarely the loudest — they are the ones that keep paying rent.
The Problem
Every year brings a new “must-have” tool. Some are genuinely useful. Some solve a short-term problem. And a few quietly earn a permanent place because they keep solving the same boring, recurring work well.
The tools that did not stick were usually the ones that required significant setup without enough repeated value — or the ones that solved a problem I did not actually have often enough. Choosing tools by novelty is a reliable way to spend a year on setup instead of on the work.
The Approach
Over 10 years, three tools have stayed in my stack. Each one earns its place the same way: it solves a recurring problem with low friction.
SQL
Still the fastest way to test an idea close to the data. Before building a notebook, feature pipeline, dashboard, or model, a query often gives the first useful answer.
LightGBM / XGBoost
For many structured tabular problems, gradient-boosted trees remain serious baselines. Not because they are trendy, but because they are efficient, mature, and often difficult to beat without a clear reason for more complexity.
Pydantic
Earned its place by making assumptions explicit at system boundaries. API inputs, configuration, pipeline handoffs, and structured outputs become easier to validate, review, and maintain.
The pattern is simple: these tools help you inspect data, establish a reliable baseline, define a clear contract, or ship something maintainable. Each one covers a repeated, boring task that otherwise costs real time.
Outcome
The longer I work, the less I optimise for novelty. I optimise for tools I still reach for when there is real work to do. That shift is what turns a pile of libraries into a working stack.
The most durable tools are rarely the loudest. They are the ones that keep paying rent.
Key Takeaway
Design insight: The tools that survive solve recurring problems with low friction. After 10 years in data science, the tools that stayed were rarely the flashiest ones — they were the ones that keep paying rent. SQL still gives the first useful answer because it tests an idea close to the data, before a notebook, feature pipeline, dashboard, or model exists. LightGBM and XGBoost remain serious baselines for structured tabular problems because they are efficient, mature, and hard to beat without a clear reason for more complexity. Pydantic holds its place by making assumptions explicit at system boundaries, so API inputs, configuration, pipeline handoffs, and structured outputs are easier to validate, review, and maintain. What did not stick were tools that required significant setup without enough repeated value, or that solved a problem rarely encountered. The longer you work, the less you optimise for novelty and the more you optimise for tools you still reach for when there is real work to do. The most durable tools are rarely the loudest — they are the ones that keep paying rent.