Data-Engineering

Introduction In this blog post, we’ll zoom into the details of how column ordering and data types can cause issues when using the union function in Apache Spark to combine two dataframes. We’ll explore real-world examples that illustrate the problem and provide practical solutions to overcome these challenges. By the end of this post, you’ll have a better understanding of how to use union effectively and avoid common pitfalls that can lead to job failures. ...

Introduction Hi there 👋! In this blog post, we will explore why JSON is not suitable as a big data file format. We’ll compare it to the widely used Parquet format and dig deep to demonstrate, through examples, how the JSON format can significantly degrade the performance of your data processing jobs. JSON (JavaScript Object Notation) is a popular and versatile data format, but it has limitations when dealing with large-scale data operations. On the other hand, Parquet, an open-source columnar storage format, has become the go-to choice for big data applications. ...

Introduction Welcome, readers! 📖 In this post, we’ll explore the idea of event source mapping in AWS, with a focus on its implementation and functionality. We’ll zoom-in 🔍 how automatic scaling works and examine the process of consuming messages from Kafka event sources. Using Lambda to consume records from Kafka Processing streaming data with traditional server-based technologies and Kafka consumers written in Scala can often introduce unnecessary overhead for simple tasks like creating custom sink consumers to save or delete data based on specific rules. In such scenarios, where complex data processing, joining multiple topics, or a full-fledged streaming application isn’t required, a Python-based Lambda function emerges as a viable alternative. ...

Introduction Hi, welcome to my first blog post. This post is the first one in a series of many that will follow. Who is Tortoise and who is Hare? Well, In many books about Apache Spark that I was reading, I didn’t found a clear idea of the performance of dataframes compared to the datasets. In this blog post, we will debunk that mystery and show some concrete results and insights regarding this matter. ...