The Goal of Having a Validation Practice:
Whenever a company introduces a new technology to an industry, it comes with a need for an intense validation practice to support, with data, that the technology produces real results. The technology around the hypersensing techniques that we use is one of the greatest innovations in the weather industry in the last decade. We feel it’s our duty to get it right. That’s why, at Tomorrow.io, we have made it a top priority to build a validation practice made up of top meteorologists from MIT and other leading scientific institutions, to constantly monitor results and optimize our solution. The validation leadership team is listed below.
Validation Leadership Team:
How Our Validation Process Works:
The validation team has the primary mission of building a repeatable process for measuring and validating the weather data that we produce. At the center of the process is the comparison of three sets of data:
- Data from Tomorrow.io
- Data from a comparable source for a specific weather category (for example NEXRAD radar data for precipitation)
- Data from a source considered to be the closest to ‘ground truth’ (for example, rain gauge data for precipitation) for a record of what actual weather conditions were present at a given time and location
Using these three sources, the team has built a deep set of analyses around how data from Tomorrow.io and another comparable source correlate to the ground truth for a certain event. For example, in a given precipitation event at a specific location, Tomorrow.io may have observed that the rain started at 3:42pm, NEXRAD radar may have observed that it started at 4:02, and the ‘ground truth’ from the rain gauge may have observed that it started at 3:44 or perhaps that it didn’t rain at all. From these three sets of data we can see patterns of where Tomorrow.io has advantages over traditional technologies, and where both technologies still have areas for improvement. In the company’s early days, this analysis was done through detailed spreadsheet calculations fed from these multiple data sets. Recently, Tomorrow.io finished building a comprehensive online validation tool that automates the process and empowers the team to quickly change date ranges and other criteria to see where there are discrepancies. Below are some more details about this tool.
Tomorrow.io’s validation tool is a comprehensive software tool comprised of our proprietary data, data from a second comparable source for each weather category, and data that represents the ‘ground truth’ observation of what really happened. It includes visualization tools to track differences over time and is how we analyze the performance of Tomorrow.io data against ground truth information as well as data from a comparable measurement tool. For the precipitation data that is analyzed, a detailed picture of timing performance is shown through the usage of categorical statistics, which are entirely based on binary precipitation detection data.
What We Measure:
Currently, the validation practice at Tomorrow.io measures the performance of our data in three categories of weather:
- Nowcast (short term forecast of precipitation specifically)
For each of these weather categories, the team has set up analyses around the following measures:
- False Alarm Rates
The analysis done by the validation team thus far, both manually and with our automated validation tool, has delivered the following conclusions:
- Tomorrow.io’s proprietary hyper-sensing technologies are as good, and in most cases far better, than the comparable industry standard technology for sensing precipitation and temperature and at making short-term forecasts (nowcasts).
- When using the Tomorrow.io technologies together with traditional technologies, we gain the ability to build a fused weather data engine, that contains insights not available from any other organization because of the proprietary hyper-sensing data sources exclusive to Tomorrow.io.
- We have also been able to improve the quality of our tools based on the validation team’s findings and will continually improve the technology and models going forward.
Our validation team has built detailed report summaries containing the findings of their research for each weather category. Download the reports here.
For any questions or additional parameters statements, please email email@example.com or click on the chat window in the bottom right.