Report: Data engineers spend 2 days a week fighting the fire of poor data quality

Previous industry surveys have shown that most data professionals struggle with poor data quality, despite the amount of time and resources invested in fixing it. To extend these insights and help data leaders understand where and how to invest resources, data reliability companies, Monte Carlopartnered with Wakefield Research to survey more than 300 data professionals to reveal the state of data quality in the broader market.

The results from the study show that teams are still grappling with data quality and wasting valuable time, money, and resources fixing issues that arise. Respondents reported that, on average, 40% of their time was spent evaluating or checking data quality. In addition, more than half of the respondents rate the construction or repair pipeline because they take up most of the day.

When a data incident occurs, the majority of data specialists – 62% – say their detection time is 5 to 8 hours. The average time to resolve data issues is nine hours.

Data quality and bottom line

Poor data quality also affects profits – with 47% of respondents estimating that bad data affects 25% or more of their company’s revenue.

Of the 300 data professionals surveyed:

• 75% take four hours or more to detect data quality issues.
• About half said it took an average of 9 hours to resolve the issue once it was identified.
• 58% said the total number of incidents had increased somewhat or greatly over the past year, often due to more complex pipelines, larger data pools, larger data volumes, and other factors.

However, there is some positive news. About 90% of respondents said they have invested in or plan to invest in data quality solutions, such as data observabilitywithin six months.

The survey of more than 300 data professionals was conducted from April 28 to May 11, 2022, using email invitations and online surveys.

Read Full report from Monte Carlo.