Data-Processing

The Problem: Union function isn’t as Simple as it Seems Picture this: You have a large dataset that you need to process in different ways, so you: Split it into smaller pieces Transform each piece differently Put them back together using union Sounds straightforward, right? Well, there’s a catch that most developers don’t know about. The Hidden Performance Killer 🐌 Here’s what’s actually happening behind the scenes when you use union:...

DuckDB Inside Postgres: The Unlikely Duo Supercharging Analytics If you work in data engineering, you know that the field moves at a dizzying pace. New tools and technologies seem to pop up daily, each promising to revolutionize how we store, process, and analyze data. Amidst this constant change, two names have remained stalwarts: Postgres, the tried-and-true relational database, and DuckDB, the talented new kid on the block for analytics workloads....

Introduction In this blog post, we’ll zoom into the details of how column ordering and data types can cause issues when using the union function in Apache Spark to combine two dataframes. We’ll explore real-world examples that illustrate the problem and provide practical solutions to overcome these challenges. By the end of this post, you’ll have a better understanding of how to use union effectively and avoid common pitfalls that can lead to job failures....

Introduction Hi there 👋! In this blog post, we will explore why JSON is not suitable as a big data file format. We’ll compare it to the widely used Parquet format and dig deep to demonstrate, through examples, how the JSON format can significantly degrade the performance of your data processing jobs. JSON (JavaScript Object Notation) is a popular and versatile data format, but it has limitations when dealing with large-scale data operations....