#data-engineering

[ follow ]
#data-analytics #cloud-computing #data-quality #ai-solutions #generative-ai #e-commerce #apache-spark #scalability
fromInfoQ
1 week ago

How Netflix Powers Audience Insights at Trillion-Row Scale

Muse helps creative and launch teams at Netflix understand which artwork and video assets resonate with audiences, and its growth required supporting advanced filtering and audience-affinity analysis at massive scale. To meet these demands, Netflix reports that it redesigned the data serving layer, cutting query latencies by about 50% while maintaining accuracy and responsiveness. Muse started as a Spark-powered dashboard backed by a modest Apache Druid cluster.
Data science
#data-quality
more#data-quality
Startup companies
fromTechzine Global
4 weeks ago

Fivetran strengthens data platform through acquisition of Tobiko Data

Fivetran acquired Tobiko Data to expand transformation capabilities and deliver faster, controlled, AI-ready data with an open, interoperable foundation for transparency and scalability.
Data science
fromInfoWorld
1 year ago

What is Microsoft Fabric? A big tech stack for big data

Microsoft Fabric is an integrated SaaS platform that combines various data analytics tools into a unified cloud-based environment.
Data science
fromComputerWeekly.com
4 weeks ago

Interview: Holland & Barrett CDO preparing a Michelin star-worthy data strategy | Computer Weekly

Holland & Barrett models its data organisation on a three‑Michelin‑star restaurant: raw data foundations, a data warehouse to process data, and analysts serving insights.
Data science
frommedium.com
1 month ago

Complete Guide to Learn Big Data

Learn big data end-to-end: fundamentals, programming, storage, batch/stream processing, ETL, cloud, ML, governance, and hands-on projects with runnable Airflow and PySpark Docker examples.
Data science
fromMedium
1 month ago

Building Resilient Data Systems: Key Lessons from Veronika Durgin

Neglected data engineering tasks are crucial for stable and agile data pipelines.
Data science
fromInfoWorld
1 month ago

Google updates agents in BigQuery to further automate analytics tasks

Google enhances BigQuery with a new code interpreter and advanced analytics features, improving automation in data engineering and data science tasks.
Artificial intelligence
fromInfoQ
2 months ago

Mandy Gu on Generative AI (GenAI) Implementation, User Profiles and Adoption of LLMs

Generative AI and large language models are being implemented in real-world projects to enhance organizational capabilities.
fromHackernoon
2 years ago

The HackerNoon Newsletter: A Data Engineers Guide to PyIceberg (7/6/2025) | HackerNoon

The arrival of truly intelligent, always-on, AI-native revenue engines is dismantling the way we've structured go-to-market motions for 20 years.
Tech industry
fromInfoWorld
3 months ago

Databricks targets AI bottlenecks with Lakeflow Designer

Lakeflow and OpenFlow reflect two philosophies: Databricks integrates data engineering into a Spark-native, open orchestration fabric, while Snowflake's OpenFlow offers declarative workflow control.
Software development
Data science
fromTheregister
3 months ago

Industry reacts to DuckDB's Lakehouse architecture reorg

Databricks' acquisition of Tabular is revitalizing the table formats landscape, especially with DuckDB's innovative offerings.
Data science
fromInfoWorld
3 months ago

Snowflake launches Openflow to tackle AI-era data ingestion challenges

Openflow simplifies data ingestion, transformation, and observability for enterprises engaging with AI use cases.
fromTechzine Global
4 months ago

Fivetran expands Connector SDK for custom data sources

Fivetran's Connector SDK empowers developers to create custom connectors easily, addressing data gaps and enabling centralized data management without the need for extensive DevOps support.
Data science
#e-commerce
fromHackernoon
5 months ago
Data science

Rajesh Sura: Revolutionizing Global Selection Strategy with Data, AI, and Automation | HackerNoon

Product selection is crucial for competitive advantage in e-commerce.
Rajesh Sura's innovations in data engineering have transformed product onboarding and sourcing.
AI-driven models and external data integration enhance product discovery.
frommedium.com
4 months ago
Data science

Day 4Identifying Top 3 Selling Products per Category | Spark Interview Question.

Implement logic to rank products per category based on total units sold, ensuring ties are handled appropriately.
fromHackernoon
5 months ago
Data science

Rajesh Sura: Revolutionizing Global Selection Strategy with Data, AI, and Automation | HackerNoon

more#e-commerce
#apache-spark
fromMedium
4 months ago
Data science

Day 6-Sessionization of Web Logs using Time Difference | Apache Spark Interview Problem.

fromMedium
4 months ago
Data science

Understanding the load() Function in Apache Spark: Syntax, Examples, and Best Practices

fromMedium
4 months ago
Data science

Day 6-Sessionization of Web Logs using Time Difference | Apache Spark Interview Problem.

fromMedium
4 months ago
Data science

Understanding the load() Function in Apache Spark: Syntax, Examples, and Best Practices

more#apache-spark
Data science
fromHackernoon
5 months ago

How Bharath Rajasekaran Scaled a Global Data Pipeline in 3 Months | HackerNoon

Annalect's data pipeline migration exemplifies outstanding engineering leadership and has transformed the company's approach to cloud data management.
fromMedium
4 months ago

Day 3-Revenue Aggregation per Region and Category | Spark Interview Problem.

To create a daily aggregated revenue dashboard, we need to sum the total revenue for each product category by region, helping managers make informed business decisions.
Data science
fromHackernoon
5 years ago

Traditional Monitoring Is Dead. Long Live Data Observability | HackerNoon

Traditional monitoring fails to meet the needs of complex data organizations; instead, engineers must develop interactive observability frameworks to quickly identify anomalies.
Data science
fromHackernoon
3 years ago

Building a Real-Time Change Data Capture Pipeline with Debezium, Kafka, and PostgreSQL | HackerNoon

The article provides a step-by-step guide to setting up a Change Data Capture (CDC) pipeline using PostgreSQL, Debezium, Apache Kafka, and Python.
Data science
Data science
fromHackernoon
8 months ago

LLMs in Data Engineering: Not Just Hype, Here's What's Real | HackerNoon

Large Language Models are transforming data engineering by enhancing performance and operational efficiencies.
DevOps
fromMedium
5 months ago

Evolvability-It's Mostly About Data Contracts

Data Contracts can mitigate complexity in analytic systems by fostering loose coupling and enhancing adaptability.
Data science
fromHackernoon
5 months ago

Tired of Copy-Pasting Hive Output? This PySpark Hack Fixes It | HackerNoon

Automating CSV export from Hive or Impala output is essential for efficient data engineering tasks.
Women in technology
fromBusiness Insider
5 months ago

I became a director at Ford after pivoting careers in the last recession. Here are 3 ways to recession-proof your job.

Continuous learning through online courses is key to job security in recessionary times.
fromChannelPro
5 months ago

Datatonic expands global services with Syntio acquisition

We are thrilled to welcome Syntio to the Datatonic family. This acquisition is a key step in our strategy to expand our global reach and enhance our service capabilities.
Data science
Data science
fromTechzine Global
5 months ago

Datatonic acquires Syntio and strengthens expertise in data engineering

Datatonic's acquisition of Syntio enhances its data consultancy with increased capabilities in data engineering and expanded service offerings.
Data science
fromawstip.com
5 months ago

Spark Scala Exercise 23: Working with Delta Lake in Spark ScalaACID, Time Travel, and Upserts

Delta Lake enhances data reliability and governance for data lakes by integrating warehouse features.
[ Load more ]