Category: Big Data

As organizations deploy Amazon Managed Streaming for Apache Kafka (Amazon MSK) clusters across multiple use cases, the manual management of topic configurations can be challenging. This can lead to several issues: Inefficiency – Manual configuration is time-consuming and error-prone, especially for large deployments. Maintaining consistency across multiple configurations can be difficult. To avoid this, Kafka…

Read More “Automate topic provisioning and configuration using Terraform with Amazon MSK” »

This post is co-written by Dr. Leonard Heilig and Meliena Zlotos from EUROGATE. For container terminal operators, data-driven decision-making and efficient data sharing are vital to optimizing operations and boosting supply chain efficiency. Internally, making data accessible and fostering cross-departmental processing through advanced analytics and data science enhances information use and decision-making, leading to better…

Read More “How EUROGATE established a data mesh architecture using Amazon DataZone” »

This post is cowritten by Ishan Gupta, Co-Founder and Chief Technology Officer, Juicebox. Juicebox is an AI-powered talent sourcing search engine, using advanced natural language models to help recruiters identify the best candidates from a vast dataset of over 800 million profiles. At the core of this functionality is Amazon OpenSearch Service, which provides the…

Read More “Juicebox recruits Amazon OpenSearch Service for improved talent search” »

Organizations constantly work to process and analyze vast volumes of data to derive actionable insights. Effective data ingestion and search capabilities have become essential for use cases like log analytics, application search, and enterprise search. These use cases demand a robust pipeline that can handle high data volumes and enable efficient data exploration. Apache Spark,…

Read More “Batch data ingestion into Amazon OpenSearch Service using AWS Glue” »

In our previous post Backtesting index rebalancing arbitrage with Amazon EMR and Apache Iceberg, we showed how to use Apache Iceberg in the context of strategy backtesting. In this post, we focus on data management implementation options such as accessing data directly in Amazon Simple Storage Service (Amazon S3), using popular data formats like Parquet,…

Read More “Build a high-performance quant research platform with Apache Iceberg” »

The rise of generative AI applications has heightened the necessity to implement semantic search and natural language search. These advanced search features help find and retrieve conceptually relevant documents from enterprise content repositories to serve as prompts for generative AI models. Raw data within various source repositories in the form of text, images, audio, video,…

Read More “Cost Optimized Vector Database: Introduction to Amazon OpenSearch Service quantization techniques” »

Quick and reliable access to information is crucial for making smart business decisions. That’s why companies are turning to Amazon OpenSearch Service to power their search and analytics capabilities. OpenSearch Service makes it straightforward to deploy, operate, and scale search systems in the cloud, enabling use cases like log analysis, application monitoring, and website search….

Read More “Use CI/CD best practices to automate Amazon OpenSearch Service cluster management operations” »

Google Analytics 4 (GA4) provides valuable insights into user behavior across websites and apps. But what if you need to combine GA4 data with other sources or perform deeper analysis? That’s where Amazon Redshift and Amazon AppFlow come in. Amazon AppFlow bridges the gap between Google applications and Amazon Redshift, empowering organizations to unlock deeper…

Read More “Ingest data from Google Analytics 4 and Google Sheets to Amazon Redshift using Amazon AppFlow” »

The Amazon EMR runtime for Apache Spark offers a high-performance runtime environment while maintaining 100% API compatibility with open source Apache Spark and Apache Iceberg table format. Amazon EMR on EC2, Amazon EMR Serverless, Amazon EMR on Amazon EKS, Amazon EMR on AWS Outposts and AWS Glue all use the optimized runtimes. In this post,…

Read More “Amazon EMR 7.5 runtime for Apache Spark and Iceberg can run Spark workloads 3.6 times faster than Spark 3.5.3 and Iceberg 1.6.1” »

Real-time data streaming and event processing are critical components of modern distributed systems architectures. Apache Kafka has emerged as a leading platform for building real-time data pipelines and enabling asynchronous communication between microservices and applications. However, running and managing Kafka clusters at scale can be challenging, requiring specialized expertise and significant operational overhead. Amazon Managed…

Read More “Fitch Group achieves multi-Region resiliency for mission-critical Kafka infrastructure with Amazon MSK Replicator” »