From the Wall Street Journal, May 23:
On a recent Sunday evening, Samir Madani had dinner with his family 
in suburban Stockholm, did the dishes and put his two children to bed.
Then he opened his laptop and started crunching U.S. oil import data late into the night.
Mr. Madani, a technology executive who trades and researches crude as
 a hobby, is part of a growing group of oil sleuths who have sprung up 
to sate the market’s voracious appetite for data and intelligence.
“So much of oil data is hidden and we’re trying to make it 
accessible,” said Mr. Madani, who runs a free oil data website from his 
house. “Besides, there’s so much drama in oil.”
Dramatic gyrations in the price of oil in the past three years have 
boosted demand for such services, intensifying competition in a market 
that for years had been dominated by governments, oil companies and a 
handful of big data providers.
Oil Sleuths
Oil data gatherers use a
 variety of technologies to match the market’s voracious appetite for 
statistics and scraps of intelligence.
Satellites
• Companies use satellites to monitor oil tankers as they criss-cross the seas
• That helps them reveal the location of Iranian oil tankers or how congested major oil ports
• Satellites which can deliver more frequent imagery than traditional satellites  
Infrared cameras
• Louisville, Ky.-based Genscape uses infrared cameras to measure the level of stored oil
• Those cameras are mounted on helicopters that fly around tanks around the world
• The market-moving data is usually released a few days ahead of government statistics  
 
 
Computer algorithms
• Oil data firms use computer programs to analyze reams of data to come up with forecasts for key oil numbers
• Input sources include: satellite imagery, customs databases, shipping records, social news
• The data is then distributed via online portals featuring charts and live tanker tracking maps
Source: the companies; staff reports
 
The new entrants include both amateurs armed with an internet connection and a Twitter
 account, and professional services using shoebox-sized satellites and 
sophisticated computer models. They are crunching data on everything 
from Middle Eastern exports to U.S. drilling. Such statistics often 
move oil prices as they predict government releases on crude inventories
 or cover data black spots such as Chinese stockpiles and Iranian tanker
 movements.
With the proliferation of data sources, the oil industry is catching 
up to other sectors. Retail and commodity investors, for instance, have 
long had access to a wealth of sophisticated information on things like 
store traffic and crop yields. But the free-fall in the price of 
oil—from more than $100 a barrel in 2014 to around $50 today—created new
 trading opportunities for hedge funds and day traders.
When Doug King started the Merchant Commodity hedge fund at RCMA 
Asset Management in 2004, there were only a few outside data sources, he
 says. “We used to do our own data crunching by hand, it was a simpler 
time,” said Mr. King, who now subscribes to several data services. 
“The oil data industry has exploded,” said Mr. King, chief investment officer at RCMA.
At least three new oil data services companies are launching this year.
One is Kayrros, a Paris-based startup due to open for business in 
June. It aims to use computer algorithms to analyze satellite imagery, 
financial data and social news to come up with detailed estimates and 
forecasts for key oil numbers, says Antoine Halff, a founding partner of
 the firm.
“A decade or more ago, it used to be people with binoculars sitting 
in a hotel watching tankers move in and out of the port,” said Mr. 
Halff, who is also a senior researcher at Columbia University’s Center 
on Global Energy Policy.
A popular service such firms offer is to track where oil tankers are 
going. That gives, for instance, insight into how much crude that 
members of the Organization of the Petroleum Exporting Countries are 
exporting after their deal last year to limit supply....MUCH MORE