Big Data Analysis Dashboard

Challenge

With the increasing connectivity of vehicles and the use of telemetry, the automotive industry is generating vast amounts of data. Our client faced the challenge of processing vehicle data for more than 8 million vehicles across over 100 markets and more than 200 billion data points, and making it available for further use within the company via a dashboard. The dashboard needed to be performant, flexible in use, and easily scalable as usage increased. An anticipated increase in data volume also had to be considered.

Approach

To meet the various requirements of the dashboard, we decided to implement the entire application as a containerized microservice architecture in the cloud, dividing it into a powerful backend and a flexibly scalable frontend. The backend runs on an Apache Spark cluster and can automatically scale its computing capacity upwards as demand increases. On weekends, holidays, and after extended inactivity, the cluster shuts down cost-effectively and only reactivates upon incoming data queries and shortly before work begins. The frontend was developed with R Shiny and is based on containerized images, allowing it to be deployed on different platforms like Docker or Kubernetes and to easily accommodate increasing numbers of users.

Result

By integrating innovative cloud technologies, we successfully brought a modern big data analysis dashboard into production for our client. The dashboard is now actively used by over 300 users within the company. The ongoing integration of new telemetry data and the resulting new features promise to continue delivering significant value for our client in the future.