stn 03 · quantil — client engagement
Forecasting Energy Demand Across Three Planning Horizons
Built the model that turns a production plan into the electricity it will need — across ten operating units, three planning horizons, and six years of daily data that had never been assembled in one place.
sec 01 · brief
The problem
A national oil and gas operator plans production on three clocks at once: an operating plan three months out, a financial plan three years out, and a long-range plan ten years out. Every one of those plans has to carry an electricity figure beside its production figure, because the fields both buy power and generate their own, and the balance between the two is a capital decision.
Those electricity figures were being produced without a model. The data to build one existed, but not in one place: six years of daily consumption metered per frontier, six years of daily autogeneration in a different file format, production in three mutually incompatible layouts across three overlapping sources, water reported monthly while everything else was daily, and no maintained correspondence between the frontiers that meter the energy and the units that own the fields. Thirty frontiers carried no field assignment at all.
The warehouse that was supposed to hold this was rejected on inspection as incomplete and out of date, which set the real shape of the work: before anything could be forecast, six years of raw monthly files had to be reconciled into one daily series per operating unit.
sec 02 · approach
What I built
Supervised regression per operating unit, not a time-series forecast. The engagement opened with sequence models over the energy history as the intended approach, and that approach needs a unified, regularly maintained base integrating production and energy — which is exactly the thing that did not exist. Regressing energy on the production variables sidesteps the missing history entirely: a planning document already states the crude, gas, light-hydrocarbon and water volumes it assumes, so the model is handed its inputs and never has to extrapolate its own past. That also means one fitted model serves all three horizons, because the horizon lives in the input file rather than in the model.
Two models per unit rather than one, because the short-horizon operating plan carries no water column and the two longer plans do, and water turned out to carry most of the signal. Twelve candidates per unit — four feature sets crossed with three algorithms — selected on lowest MAE, with the best water model and the best no-water model both kept. Each ships beside its own fitted preprocessing pipeline, so the transformations applied at inference are the ones applied at training rather than a reimplementation of them.
sec 03 · method
Method
01
Source reconciliation
Eleven source files spanning 2018–2024. Energy arrives twice: consumption, as one file per month per frontier, filtered to production frontiers so transport and refining are excluded; and autogeneration, as a single concatenated series with every non-generating frontier dropped. Both are mapped to operating units through a correspondence table built for the purpose, because frontier, field and unit names disagree across sources and thirty frontiers have no field assigned at all — which is what makes the unit, not the field, the only level everything can be joined at. Three units meter autogeneration and no purchased power.
02
Unit conversion and alignment
Production comes in three layouts. The 2018–2021 files needed only header normalisation; the 2022–2024 gross files are in thousands of barrels of oil equivalent per day and are multiplied by 1,000, then reshaped wide-to-long. Where the gross files and the consolidated 2020–2024 file overlap they disagree, and the maximum of the two is taken as the reconciled value. Water is reported monthly, so each month's volume is divided across its days and converted from cubic metres at 6.289 barrels per cubic metre — which leaves a visibly stepped daily series, and that step is an artifact of the source's granularity rather than a property of the field.
03
Outlier removal and interpolation
Two custom scikit-learn transformers, so the same treatment applies identically at training and at inference. The first flags values beyond three standard deviations computed per unit per quarter — quarterly rather than globally, because a six-year series with real regime changes in it would otherwise have entire true periods flagged — and for energy it treats an exact zero as an outlier too, since a metered zero is a reporting gap rather than a day without power. Flagged values become null. The second reindexes each unit-quarter onto a continuous date range and interpolates on time. Energy and light hydrocarbons needed the most correction; one unit's energy series required it on roughly 19.5% of its records against approximately 5% everywhere else.
04
Feature sets and model selection
Four feature sets per unit — water present or absent, crossed with cyclical day-of-year encodings present or absent, the latter as the sine and cosine of 2π·day/365 so that December 31 and January 1 sit next to each other instead of at opposite ends of a line. Three algorithms over each: linear regression, random forest, gradient boosting. Twelve candidates per unit, 120 in all, selected on lowest MAE. Two survive per unit — the best with water and the best without — for 20 shipped models and 20 shipped pipelines.
05
Horizon-specific inference
The same fitted models serve all three horizons; only the input file changes. Under a year, monthly, from the operating plan, using the no-water models. One to three years, monthly, from the financial plan, using the water models. Ten years, annual, from the long-range plan — where per-project estimation was tested and abandoned, because individual project volumes sit far below anything in the training data and the models are not calibrated at that magnitude. Instead the unit is predicted as a whole and its energy apportioned to projects by a production-weighted share, which sums to one by construction.
06
Delivery and retraining
A web tool ingests Excel or CSV, validates it — required columns present, no duplicate column names, no nulls, no negatives after grouping, dates well-formed, at least half the rows populated, and a row count consecutive and consistent with the declared horizon — and returns two formats: the input with a predicted-kWh column appended, and a planning-tool export at vice-presidency level in megawatts carrying medium, high and low scenarios, where high and low are the prediction plus and minus one standard deviation of that unit's training data. Retraining stays in three notebooks — preprocessing, per-unit training, best-model selection — so the team can inspect the charts behind a selection rather than accept it.
sec 04 · legend
Instruments
legend · modelling
legend · preprocessing
legend · platform
legend · delivery
sec 05 · readings
Results
Models trained
120
10 units × 4 feature sets × 3 algorithms
Models shipped
20
2 per unit — with and without water
Monthly MAPE, with water
0.18 – 1.76 %
10 of 10 units
Monthly R², with water
0.56 – 1.00
9 of 10 at 0.97 or above
Daily MAPE, with water
1.30 – 14.67 %
test split · 8 of 10 units below 5 %
Selected family
RF 6 · XGB 4 · OLS 0
10 water models, lowest MAE
Training span
2018 – 2024
daily series, 10 units
Output formats delivered
2 of 3
dispatch-level deferred
plate 01
Monthly error, water models
BARITE
BASALT
GALENA
GYPSUM
JASPER
OLIVINE
PYRITE
QUARTZ
RUTILE
ZIRCON
Mean absolute percentage error by operating unit, on the monthly aggregation the financial and long-range plans actually consume. Every unit lands under 2%, and the spread between the best and the worst is a factor of ten — which is the more useful reading, because it says the model is not equally good everywhere even where it is good everywhere.
source
Selected water model per unit, daily predictions aggregated by month. Unit names are pseudonyms; the figures are the real ones.
plate 02
Dropping water breaks exactly one unit
BARITE
BASALT
GALENA
GYPSUM
JASPER
OLIVINE
PYRITE
QUARTZ
RUTILE
ZIRCON
The same measure for the no-water models, which are the ones the short-horizon operating plan has to use because it states no water volumes. Nine units absorb the loss and stay under 2.4%. One does not, and it fails by a factor of eleven against its own water model — so the cost of dropping water is not a general tax on accuracy but a single localised break.
source
Selected no-water model per unit, daily predictions aggregated by month. The axis runs to 26 to hold the outlier, which is why nine bars read as slivers — every bar is direct-labelled for exactly that reason.
plate 03
Daily error is where the models are tested
BARITE
BASALT
GALENA
GYPSUM
JASPER
OLIVINE
PYRITE
QUARTZ
RUTILE
ZIRCON
The same water models scored at the daily resolution they were actually fit at, on held-out data. Eight of ten units stay under 5%. Monthly aggregation is flattering by construction — daily errors cancel within a month — so this is the honest picture of the models, and plate 01 is the picture of the product.
source
Test split only. The train split is lower for every unit without exception and is deliberately not plotted beside it, since showing both invites the flattering number to be read as the result.
plate 04
Nine units are explained; one is not
BARITE
BASALT
GALENA
GYPSUM
JASPER
OLIVINE
PYRITE
QUARTZ
RUTILE
ZIRCON
Coefficient of determination for the same monthly water models — the share of each unit's energy variation the production variables account for. Nine units sit at 0.97 or above. One sits at 0.56, and it is not the unit with the worst error: a model can track a series closely without accounting for why it moves, and this plate is the only place that distinction is visible.
source
Same models and same aggregation as plate 01. A plate that is almost all one height is usually not worth drawing; here the uniformity is what makes the exception legible.
sec 07 · findings
What the data said
finding · result
Monthly aggregation is where the tool is accurate
Every unit's monthly error with water sits between 0.18% and 1.76%, and nine of ten monthly R² values are 0.97 or above. This is the number the planning cycles actually consume — the plans are monthly and annual documents — and it is the number the tool was built to produce.
finding · result
Water carries the model, and so does the calendar
All ten selected water models use the feature set that includes both water and the cyclical day-of-year encodings; no variant lacking the cyclical features was ever selected, for any unit, under any algorithm. Dropping water costs most units under one percentage point of monthly error, which is what makes a usable short-horizon model possible at all — the operating plan states no water.
finding · negative result
One unit changed regime mid-window and no model survived it
RUTILE is the worst unit on every plate: 1.75% monthly with water, 25.51% without, 14.67% daily on the test split with water and 41.21% without. Its energy series runs at 1.0–2.15 million kWh through mid-2021, then falls to a flat ~0.07 million and stays there. The report attributes this to its energy frontiers being distributed across other units rather than to anything about the field, which makes it a data-model problem and not a modelling one.
finding · negative result
One unit's energy is not explained by its production
PYRITE returns R² 0.56 with water and 0.60 without — the only unit below 0.9 on either, while every other unit sits at 0.96 or above. It also carries the lowest daily test error on the sheet at 1.30%. Low error and low explained variance together describe a series the model tracks closely and whose movement it does not account for. The source does not reconcile the two, and neither does this page.
finding · negative result
Sequence models were abandoned before they were tried
The engagement opened intending recurrent forecasting over the energy history. It was dropped on the finding that no unified, regularly updated base integrating production and energy existed to train one on, and the work re-planned around supervised regression tolerant of fragmented, uneven data. Building that base is named in the report as the prerequisite for any advanced method later.
finding · negative result
The warehouse was unusable and the raw files won
The energy data warehouse was the intended source and was rejected on inspection as incomplete and out of date. Six years of per-month raw frontier files were parsed instead, which is most of the first two method stages and most of the engagement's risk.
finding · negative result
The dispatch-level output could not be built
Three output formats were designed and two shipped. The third distributes demand to individual electrical nodes, which requires a field-to-node correspondence — and the most recent correspondence file covers 39 fields against 103 in the training data. Most fields could not be mapped, so the format was deferred rather than shipped partially populated.
finding · negative result
The long-horizon allocation is proven but not wired in
The production-weighted share that apportions a unit's predicted energy to individual projects was validated against the data and, at the close of the MVP, had not been integrated into the tool. The ten-year path therefore runs at unit level in the delivered product.
sec 08 · forward
Recommendations
01
Consolidate the dispersed sources into one maintained base. The report names this as the prerequisite for sequence models, and it is also what would collapse the first two method stages from most of the engagement into an ingestion step.
02
Extend the field-to-node correspondence past 39 of 103 fields. That single table is the only thing standing between the current output and the third format.
03
Finish wiring the long-horizon allocation into the tool, so the ten-year path returns project detail rather than unit totals.
04
Re-map the unit whose frontiers sit under other units. Its regime change is an artifact of the correspondence table, so fixing the table is what fixes the model — retraining alone will not.
05
Find a driver for the unit production does not explain, or report it with an interval. An R² of 0.56 against 0.97 everywhere else says a variable is missing; until it is found, that unit's figure should carry a band rather than a point.
sec 09 · notes
Notes
disclosure
The client is described by category and never named. The ten operating units on this page are pseudonyms; their real internal codes and names, the field names, the individuals, the internal systems and the document locations do not appear here in any form and were dropped rather than renamed. Every quantitative figure — unit counts, model counts, spans, error and fit — is the real one from the engagement, unchanged.
limits
- Models were fit at daily resolution only. The monthly figures are daily predictions aggregated by month and unit, not a model re-split and re-fit at monthly resolution — the report states a re-split was not viable without retraining. The monthly numbers therefore inherit the daily models' behaviour and are not independent of them.
- The daily figures on this page are the test split. The train split is lower for every unit.
- Outlier rates per unit are described and never charted, because the source figure carries no data labels and the values could only be recovered by measuring pixels.
- The report's prose disagrees with its own figures in three places — a daily error digit, a count of units under 5%, and which units fall below R² 0.9. The figures were taken as authoritative, and daily R² is omitted entirely because that is the disagreement no third source can settle.
- The long-horizon project allocation and the dispatch-level output were not delivered in the tool.
- Every energy series drops to zero at the right-hand edge. That is the data cutoff, not a reading, and it was excluded from training by the zero-as-outlier rule.
- The engagement delivered an MVP. Nothing here is a claim about the system in sustained production.