Data engineering is asoftware engineering approach to the building ofdata systems, to enable the collection and usage ofdata. This data is usually used to enable subsequentanalysis anddata science, which often involvesmachine learning.[1][2] Making the data usable usually involves substantialcomputing andstorage, as well asdata processing.
Around the 1970s/1980s the terminformation engineering methodology (IEM) was created to describedatabase design and the use ofsoftware for data analysis and processing.[3] These techniques were intended to be used bydatabase administrators (DBAs) and bysystems analysts based upon an understanding of the operational processing needs of organizations for the 1980s. In particular, these techniques were meant to help bridge the gap between strategic business planning and information systems. A key early contributor (often called the "father" of information engineering methodology) was the AustralianClive Finkelstein, who wrote several articles about it between 1976 and 1980, and also co-authored an influentialSavant Institute report on it with James Martin.[4][5][6] Over the next few years, Finkelstein continued work in a more business-driven direction, which was intended to address a rapidly changing business environment; Martin continued work in a more data processing-driven direction. From 1983 to 1987, Charles M. Richter, guided by Clive Finkelstein, played a significant role in revamping IEM as well as helping to design the IEM software product (user data), which helped automate IEM.
In the early 2000s, the data and data tooling was generally held by theinformation technology (IT) teams in most companies.[7] Other teams then used data for their work (e.g. reporting), and there was usually little overlap in data skillset between these parts of the business.
In the early 2010s, with the rise of theinternet, the massive increase in data volumes, velocity, and variety led to the termbig data to describe the data itself, and data-driven tech companies likeFacebook andAirbnb started using the phrase data engineer.[3][7] Due to the new scale of the data, major firms likeGoogle, Facebook,Amazon,Apple,Microsoft, andNetflix started to move away from traditionalETL and storage techniques. They started creatingdata engineering, a type ofsoftware engineering focused on data, and in particularinfrastructure,warehousing,data protection,cybersecurity,mining,modelling,processing, andmetadata management.[3][7] This change in approach was particularly focused oncloud computing.[7] Data started to be handled and used by many parts of the business, such assales andmarketing, and not just IT.[7]
High-performance computing is critical for the processing and analysis of data. One particularly widespread approach to computing for data engineering isdataflow programming, in which the computation is represented as adirected graph (dataflow graph); nodes are the operations, and edges represent the flow of data.[8] Popular implementations includeApache Spark, and thedeep learning specificTensorFlow.[8][9][10] More recent implementations, such asDifferential/Timely Dataflow, have usedincremental computing for much more efficient data processing.[8][11][12]
Data is stored in a variety of ways, one of the key deciding factors is in how the data will be used.Data engineers optimize data storage and processing systems to reduce costs. They use data compression, partitioning, and archiving.
If the data is structured and some form ofonline transaction processing is required, thendatabases are generally used.[13] Originally mostlyrelational databases were used, with strongACID transaction correctness guarantees; most relational databases useSQL for their queries. However, with the growth of data in the 2010s,NoSQL databases have also become popular since theyhorizontally scaled more easily than relational databases by giving up the ACID transaction guarantees, as well as reducing theobject-relational impedance mismatch.[14] More recently,NewSQL databases — which attempt to allow horizontal scaling while retaining ACID guarantees — have become popular.[15][16][17][18]
If the data is structured andonline analytical processing is required (but not online transaction processing), thendata warehouses are a main choice.[19] They enable data analysis, mining, andartificial intelligence on a much larger scale than databases can allow,[19] and indeed data often flow from databases into data warehouses.[20]Business analysts, data engineers, and data scientists can access data warehouses using tools such as SQL orbusiness intelligence software.[20]
Adata lake is a centralized repository for storing, processing, and securing large volumes of data. A data lake can containstructured data fromrelational databases,semi-structured data,unstructured data, andbinary data. A data lake can be created on premises or in a cloud-based environment using the services frompublic cloud vendors such asAmazon,Microsoft, orGoogle.
If the data is less structured, then often they are just stored asfiles. There are several options:
The number and variety of different data processes and storage locations can become overwhelming for users. This inspired the usage of aworkflow management system (e.g.Airflow) to allow the data tasks to be specified, created, and monitored.[23] The tasks are often specified as adirected acyclic graph (DAG).[23]
Business objectives that executives set for what's to come are characterized in key business plans, with their more noteworthy definition in tactical business plans and implementation in operational business plans. Most businesses today recognize the fundamental need to grow a business plan that follows this strategy. It is often difficult to implement these plans because of the lack of transparency at the tactical and operational degrees of organizations. This kind of planning requires feedback to allow for early correction of problems that are due to miscommunication and misinterpretation of the business plan.
The design of data systems involves several components such as architecting data platforms, and designing data stores.[24][25]
Data modeling is the analysis and representation of data requirements for an organisation. It produces a data model—an abstract representation that organises business concepts and the relationships and constraints between them. The resulting artefacts guide communication between business and technical stakeholders and inform database design.[26][27]
A common convention distinguishes three levels of models:[26]
Approaches include entity–relationship (ER) modeling for operational systems,[28] dimensional modeling for analytics and data warehousing,[29] and the use of UML class diagrams to express conceptual or logical models in general-purpose modeling tools.[30]
Well-formed data models aim to improve data quality and interoperability by applying clear naming standards, normalisation, and integrity constraints.[27][26]
A data engineer is a type of software engineer who createsbig dataETL pipelines to manage the flow of data through the organization. This makes it possible to take huge amounts of data and translate it intoinsights.[31] They are focused on the production readiness of data and things like formats, resilience, scaling, and security. Data engineers usually hail from a software engineering background and are proficient in programming languages likeJava,Python,Scala, andRust.[32][3] They will be more familiar with databases, architecture, cloud computing, andAgile software development.[3]
Data scientists are more focused on the analysis of the data, they will be more familiar withmathematics,algorithms,statistics, andmachine learning.[3][33]
| Specialties and interdisciplinarity |
|  | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Engineering education | ||||||||||||||
| Related topics | ||||||||||||||
| Glossaries | ||||||||||||||
| Other | ||||||||||||||
