Goal Of Data Engineer
- Make robust, scalable and quality data available for fact-finding and data-driven decision
- Doing data manipulation with the help of tools and technologies
Data Engineer Involves in
Collecting Data Involves
- Collecting data involves extracting, integrating and organising data from different sources
- Collecting data from databases, csv files, data lakes, data warehouse.
Processing Data Involves
- Cleaning, transforming and preparing data to make it usable
- Pipelines for extracting, transforming and loading data
Storing Data Involves
- Storing data for easy availability of data
- Make data scalable for more availability
- Ensuring data privacy, security, compliance, backups, monitoring and recovering
Making Data Available Involves
- Data available by creating Api's, services and some programs to retrieve data and pass to end user
- Checks and balance to ensure data security