AI & ML
Architectural Design Decisions for Data Ingestion in Big Data Architectures
arXiv cs.SE0 views
arXiv:2611.23297v1 Announce Type: new Abstract: Introduction: Data ingestion is the first critical step in a data architecture, in which data from various sources is moved to storage systems, such as data lakes and data warehouses, for analysis and insight generation. When developing a data ingestion pipeline, practitioners (e.g., data engineers and architects) must understand the data ingestion requirements (functional and non-functional) and select appropriate design options to support them. Objective: To help practitioners systematically develop data ingestion processes, this paper aims to compile a catalog of data ingestion design decisions, their options, and the selection drivers. Method: We conducted a systematic gray literature review on data ingestion, analyzed the selected literature using qualitative data analysis methods, and conducted semi-structured interviews with experts to assess the framework's practicality and utility. Results: From 42 gray literature articles, we identified 8 architectural design decisions, 59 decision options, and 36 decision drivers to guide practitioners in the design and implementation of data ingestion pipelines. The framework was evaluated through expert interviews, assessing its completeness, simplicity, understandability, operational feasibility, and usefulness.
Source:https://arxiv.org/abs/2611.23297PreviousAsk the Peers: Ranking NFR Coverage Gaps for Requirements ReviewNextWhen Code Works but the License Doesn't: Evaluating License Constraint Handling in Coding Agents
Related posts
![Quadruped Robot Software — Ch.12: Quadruped Case Studies [KR]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fawosw4u7jitr9h27f7c5.png)
Quadruped Robot Software — Ch.12: Quadruped Case Studies [KR]
0dev.to
How Python Is Used in Artificial Intelligence in 2026: Applications, Benefits and Career Opportunities
0dev.to

What Happens in Those Few Seconds Between Pressing the Power Button and Seeing the Windows Login Screen?
0dev.to
Getting cited by ChatGPT and Claude: what the research supports, and a free audit tool
0dev.to
Why Photoshot keeps the original color when you edit in black and white
0dev.to
ISTQB CTFL in 2027: is it still worth it? What changes, what does not
0dev.to