Data processing patterns have become a central topic in software engineering. With the growing complexity of data environments, the demand for real-time analytics, and increasing pressure to automate, the ability to design and implement effective data processing systems is now a core business requirement.
According to the 2024 State of Analytics Engineering report by dbt Labs, more than 57% of data professionals cite poor data quality as the primary barrier to effective data utilization within organizations.
Robust architecture and the application of appropriate data processing patterns not only optimize ETL/ELT workflows but also enable seamless integration with widely adopted corporate tools—such as Microsoft Excel.
In this article, we explore the evolution of architectural patterns in the context of data processing and present a practical use case demonstrating how integration with Excel can streamline real-time data access and analysis—without the need to build dedicated frontend interfaces.
Table of Contents
What Is ETL and how does it differ from ELT?
At the core of modern data processing architecture lie two key paradigms: ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform).
In the ETL model, data is:
- Extracted from various sources,
- Transformed (e.g., cleansed, normalized),
- Loaded into the target database or data warehouse.
In contrast, ELT follows a different order. Raw data is first loaded directly into data storage systems (such as data lakes), and transformation is applied only when the data is needed for analysis or consumption.
ELT is a more flexible approach, enabling organizations to work with complete, unfiltered datasets. This model supports agile data strategies, especially in environments where scalability and real-time access to raw data are crucial.
The evolution of data processing solutions
- Databases (1960s) – Pioneer: Charles Bachman
The foundation of structured data storage, enabling systematic data retrieval and management.
- Data Warehouses (1980s) – Created by Barry Devlin and Paul Murphy at IBM
- Centralized data repositories
- Historical data analytics
- High maintenance and infrastructure costs
- Data Marts – Popularized by Ralph Kimball
- Focused on specific business domains
- Enabled targeted insights
- Often resulted in data redundancy and limited organizational perspective
- Data Lakes – Introduced by James Dixon (Pentaho)
- Storage for raw, heterogeneous data
- Supported large-scale, schema-on-read processing
- Risk of becoming unmanaged “data swamps” if not governed properly
- Data Fabric – Championed by IBM and Oracle
- Unified architecture that connects diverse data sources
- Enables seamless, real-time access and governance
- Logical layer overlays physical storage for improved agility
- Data Mesh – Coined by Zhamak Dehghani (NextData, formerly ThoughtWorks)
- Domain-oriented decentralization of data ownership
- Promotes self-serve data infrastructure
- APIs serve as standardized interfaces for data consumption
- Data Lakehouse (since 2020) – Pioneered by Databricks, creators of Apache Spark
- Hybrid model combining strengths of Data Lakes and Data Warehouses
- Supports both structured and unstructured data
- Balances flexibility with governance and performance
Other common data processing patterns
- CDC (Change Data Capture)
A set of software design patterns used to detect and track changes in data so that downstream systems can act upon updated records. It enables near real-time responsiveness and efficient data synchronization across platforms. - Lambda Architecture
A hybrid model that combines batch and stream processing. It enables high accuracy through batch jobs while maintaining low-latency access to recent data via real-time streams. - Kappa Architecture
Focuses exclusively on stream processing. Designed to simplify system complexity by handling all data as a stream, whether real-time or replayed, eliminating the need for separate batch layers. - Data Vault
A data modeling methodology built to handle historical data storage from multiple operational systems. It supports scalability, auditability, and long-term agility in enterprise data warehouses.
Integration with Microsoft Excel in data processing – example use case
Although often perceived as an office tool, Microsoft Excel can become a powerful interface for interacting with data processed in backend systems. A project implemented by FireUp.pro demonstrated that:
1. The user receives an Excel plugin (an Office Add-in), which enables:
- Calling functions defined in the project (in JavaScript).
- Retrieving data from the application backend via WebSocket.
2. The configuration is based on a manifest.xml file, which defines, among other things, links to JavaScript scripts (hosted, for example, on Amazon S3).
3. The add-in:
- Is assigned to users by the Office 365 account administrator.
- Can include views, a company logo, and functions that work like regular Excel formulas (e.g., =getCustomerData(„ID123”)).
- Data is dynamically updated in the worksheet, for example, using events sent by the backend (WebSocket support).
- Communication and debugging are facilitated by the „Developer Tools” available in Excel.
Technologies and tools used in the project:
- JavaScript + Yeoman Generator for Office Add-ins
- GitHub: https://github.com/OfficeDev/generator-office
- WebSocket API for communication with the backend
- Office Add-in Manifest Reference: https://learn.microsoft.com/en-us/office/dev/add-ins/develop/add-in-manifests
Summary
Understanding data architectures and design patterns not only enables better system design for data processing but also facilitates effective integration with tools that are already well-known and widely used by business users—such as Excel.
At fireup.pro, we support such implementations by combining modern data architecture with practical process automation.
Want to implement modern data processing patterns in your company and integrate them with the tools your team uses daily, like Excel?
Get in touch with us—we’ll build an efficient solution tailored to your needs.
Learn more: https://fireup.pro/services/data-processing
