techhub.social is one of the many independent Mastodon servers you can use to participate in the fediverse.
A hub primarily for passionate technologists, but everyone is welcome

Administered by:

Server stats:

4.6K
active users

#datastreaming

0 posts0 participants0 posts today
SoftwareMill<p>Apache Kafka is a powerful tool for processing data streams - but have you ever wondered what's really happening under the hood? 🔍</p><p>You can check it with our Kafka Visualization 👉 <a href="https://softwaremill.com/kafka-visualisation/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">softwaremill.com/kafka-visuali</span><span class="invisible">sation/</span></a></p><p>With this tool, you can simulate how data flows through a replicated Kafka topic, to gain a better understanding of the message processing model. Have fun exploring Kafka!</p><p><a href="https://softwaremill.social/tags/kafka" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>kafka</span></a> <a href="https://softwaremill.social/tags/apachekafka" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>apachekafka</span></a> <a href="https://softwaremill.social/tags/kafkavisualization" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>kafkavisualization</span></a> <a href="https://softwaremill.social/tags/datastreaming" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>datastreaming</span></a></p>
Hacker News<p>Radio for DuckDB – DuckDB Now Talks to WebSockets and Redis Pub/Sub</p><p><a href="https://query.farm/duckdb_extension_radio.html" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">query.farm/duckdb_extension_ra</span><span class="invisible">dio.html</span></a></p><p><a href="https://mastodon.social/tags/HackerNews" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>HackerNews</span></a> <a href="https://mastodon.social/tags/DuckDB" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DuckDB</span></a> <a href="https://mastodon.social/tags/WebSockets" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>WebSockets</span></a> <a href="https://mastodon.social/tags/Redis" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Redis</span></a> <a href="https://mastodon.social/tags/PubSub" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>PubSub</span></a> <a href="https://mastodon.social/tags/DataStreaming" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DataStreaming</span></a> <a href="https://mastodon.social/tags/TechNews" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>TechNews</span></a></p>
InfoQ<p><a href="https://techhub.social/tags/DataStreaming" class="mention hashtag" rel="tag">#<span>DataStreaming</span></a> is tough! Despite 10+ years of attempting to simplify it, teams often spend up to 80% of their time wrangling bad data at the lake and optimizing real-time pipelines.</p><p>Discover the basic challenges of Data Streaming and a few design &amp; architecture patterns used to tackle these challenges. </p><p>This is about pragmatic solutions to build fast, scalable &amp; manageable Data Streaming Pipelines! Watch the <a href="https://techhub.social/tags/InfoQ" class="mention hashtag" rel="tag">#<span>InfoQ</span></a> video now: <a href="https://bit.ly/3Zmjrrd" target="_blank" rel="nofollow noopener" translate="no"><span class="invisible">https://</span><span class="">bit.ly/3Zmjrrd</span><span class="invisible"></span></a> </p><p><a href="https://techhub.social/tags/DataEngineering" class="mention hashtag" rel="tag">#<span>DataEngineering</span></a></p>
SoftwareMill<p>Our Client needed a robust solution to process vast amounts of data in real-time. We chose Apache Kafka for this project 🚀 See how it helped overcome all the challenges: <a href="https://softwaremill.com/case-study/data-streaming-using-kafka-for-a-retail-company/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">softwaremill.com/case-study/da</span><span class="invisible">ta-streaming-using-kafka-for-a-retail-company/</span></a></p><p><a href="https://softwaremill.social/tags/apachekafka" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>apachekafka</span></a> <a href="https://softwaremill.social/tags/kafka" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>kafka</span></a> <a href="https://softwaremill.social/tags/data" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>data</span></a> <a href="https://softwaremill.social/tags/datastreaming" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>datastreaming</span></a> <a href="https://softwaremill.social/tags/bigdata" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>bigdata</span></a></p>
Streambased<p>Kafka rocks, but has the romance faded? 😢 Is produce/consume feeling… stale?<br>Spice things up with 3 new ways to work with Kafka:<br>🔥 S3-compatible file system<br>⚡ JDBC database access<br>❄️ Apache Iceberg<br><span class="h-card" translate="no"><a href="https://mastodon.social/@streambased" class="u-url mention" rel="nofollow noopener" target="_blank">@<span>streambased</span></a></span> makes it easy.<br> <a href="https://mastodon.social/tags/ApacheKafka" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ApacheKafka</span></a> <a href="https://mastodon.social/tags/DataStreaming" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DataStreaming</span></a> <a href="https://mastodon.social/tags/DataArchitecture" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DataArchitecture</span></a></p>
zartom<p>Troubleshooting Debezium Kafka Blob Converter for DataVault Database<br>Troubleshooting your Debezium Kafka Blob Converter for efficient DataVault DB2 data streaming? This post explores common issues &amp; solutions for smooth pipeline operation. Learn to handle BLOB data effectively! <a href="https://mastodon.social/tags/DataVault" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DataVault</span></a> <a href="https://mastodon.social/tags/Debezium" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Debezium</span></a> <a href="https://mastodon.social/tags/Kafka" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Kafka</span></a> <a href="https://mastodon.social/tags/BlobConverter" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>BlobConverter</span></a> <a href="https://mastodon.social/tags/DataStreaming" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DataStreaming</span></a> <a href="https://mastodon.social/tags/DB2" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DB2</span></a><br><a href="https://tech-champion.com/database/db2luw/__trashed-22/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">tech-champion.com/database/db2</span><span class="invisible">luw/__trashed-22/</span></a><br>Resolve issues with Debezium Kafka Blob Converter when integrating with DataVault databases. Lea...</p>
PPC Land<p>Amazon debuts data streaming service for advertising metrics optimization: New service enables advertisers to access hourly performance data and campaign changes in near real-time through AWS integration. <a href="https://ppc.land/amazon-debuts-data-streaming-service-for-advertising-metrics-optimization/?utm_source=dlvr.it&amp;utm_medium=mastodon" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">ppc.land/amazon-debuts-data-st</span><span class="invisible">reaming-service-for-advertising-metrics-optimization/?utm_source=dlvr.it&amp;utm_medium=mastodon</span></a> <a href="https://mastodon.social/tags/Amazon" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Amazon</span></a> <a href="https://mastodon.social/tags/DataStreaming" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DataStreaming</span></a> <a href="https://mastodon.social/tags/Advertising" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Advertising</span></a> <a href="https://mastodon.social/tags/Marketing" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Marketing</span></a> <a href="https://mastodon.social/tags/Metrics" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Metrics</span></a></p>
Danica Fine<p>Over the past three years in <a href="https://data-folks.masto.host/tags/devRel" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>devRel</span></a>, sharing my knowledge around <a href="https://data-folks.masto.host/tags/dataStreaming" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>dataStreaming</span></a> technologies was enough to make me love everything about my job. But, we all know, it was the supportive <a href="https://data-folks.masto.host/tags/openSource" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>openSource</span></a> community that truly shaped my experience and made it the very best...</p>
A stranger in a familier land<p>It's lunch time at the datastream summit!<br>Leave the data, take the cannoli<br><a href="https://hachyderm.io/tags/datastreaming" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>datastreaming</span></a> <a href="https://hachyderm.io/tags/apacheflink" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>apacheflink</span></a> <a href="https://hachyderm.io/tags/apachepulsar" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>apachepulsar</span></a></p>
A stranger in a familier land<p><a href="https://hachyderm.io/tags/Berlin" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Berlin</span></a> and <a href="https://hachyderm.io/tags/FlinkForward" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>FlinkForward</span></a> was amazing!, <br>But I'm in <a href="https://hachyderm.io/tags/SanFrancisco" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>SanFrancisco</span></a> again for the <a href="https://hachyderm.io/tags/datastreaming" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>datastreaming</span></a> Summit next week so I'll take it! <br>The 5am <a href="https://hachyderm.io/tags/Nespresso" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Nespresso</span></a> is an admission of jetlag, nothing else....<br> <br> <br>@ververicadata @streamnativeio</p>
A stranger in a familier land<p>It's the formal opening of @FlinkFoward!<br>If you're here, the opening talks are starting!! <a href="https://hachyderm.io/tags/apacheflink" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>apacheflink</span></a> <a href="https://hachyderm.io/tags/datastreaming" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>datastreaming</span></a> <a href="https://hachyderm.io/tags/realtime" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>realtime</span></a>!</p>
The New Stack<p>The solution to long-running spans in is more creative than “don’t have long-running spans.” Learn how to address this common problem.</p><p><a href="https://hachyderm.io/tags/DataStreaming" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DataStreaming</span></a> </p><p><a href="https://thenewstack.io/opentelemetry-challenges-handling-long-running-spans/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">thenewstack.io/opentelemetry-c</span><span class="invisible">hallenges-handling-long-running-spans/</span></a></p>
The New Stack<p>Developing stateful applications can get pretty chaotic as they move upstream and downstream. 2 open source tools that aim to tame that chaos are gaining popularity.</p><p><a href="https://hachyderm.io/tags/Data" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Data</span></a> <a href="https://hachyderm.io/tags/DataStreaming" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DataStreaming</span></a> <a href="https://hachyderm.io/tags/OpenSource" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>OpenSource</span></a></p><p><a href="https://thenewstack.io/how-apache-iceberg-and-flink-can-ease-developer-pain/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">thenewstack.io/how-apache-iceb</span><span class="invisible">erg-and-flink-can-ease-developer-pain/</span></a></p>
Block Rocker<p>It was nice to see that ClickHouse has a direct integration with <span class="h-card" translate="no"><a href="https://mastodon.social/@warpstream" class="u-url mention" rel="nofollow noopener" target="_blank">@<span>warpstream</span></a></span> , so I took the opportunity to make a video that shows you just how it is done.</p><p><a href="https://mastodon.social/tags/apachekafka" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>apachekafka</span></a> <a href="https://mastodon.social/tags/dataengineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>dataengineering</span></a> <a href="https://mastodon.social/tags/datastreaming" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>datastreaming</span></a></p><p><a href="https://youtu.be/fLpwJjwZ8oA" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">youtu.be/fLpwJjwZ8oA</span><span class="invisible"></span></a></p>
Block Rocker<p>Can you consume from a <a href="https://mastodon.social/tags/ApacheKafka" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ApacheKafka</span></a> topic and write <a href="https://mastodon.social/tags/ApacheParquet" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ApacheParquet</span></a> easily? It turns out that you can, using the <a href="https://mastodon.social/tags/Bento" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Bento</span></a> open-source pipeline tool. I made a short video that shows how to consume from a <span class="h-card" translate="no"><a href="https://mastodon.social/@warpstream" class="u-url mention" rel="nofollow noopener" target="_blank">@<span>warpstream</span></a></span> topic and map and write into <a href="https://mastodon.social/tags/Parquet" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Parquet</span></a> files. I wrap it up by querying with DuckDB and then persisting it as a DuckDB database. It's fun for the whole family. </p><p><a href="https://mastodon.social/tags/dataengineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>dataengineering</span></a> <a href="https://mastodon.social/tags/datalake" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>datalake</span></a> <a href="https://mastodon.social/tags/datastreaming" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>datastreaming</span></a></p><p><a href="https://youtu.be/wurkxuorelo" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">youtu.be/wurkxuorelo</span><span class="invisible"></span></a></p>
Block Rocker<p>Is there a simple way to consume an <a href="https://mastodon.social/tags/ApacheKafka" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ApacheKafka</span></a> compatible <span class="h-card" translate="no"><a href="https://mastodon.social/@warpstream" class="u-url mention" rel="nofollow noopener" target="_blank">@<span>warpstream</span></a></span> Topic into <a href="https://mastodon.social/tags/SQLite" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>SQLite</span></a>? While there isn't a direct way to do it, I accomplished it using the open-source <a href="https://mastodon.social/tags/Bento" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Bento</span></a> pipeline project (link in the comments). With just a handful of YAML, I mapped out the inbound JSON and wrote it to a table in SQLite. Check out the goodness here:</p><p><a href="https://youtu.be/OzzcuiuF774" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">youtu.be/OzzcuiuF774</span><span class="invisible"></span></a></p><p><a href="https://mastodon.social/tags/dataengineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>dataengineering</span></a> <a href="https://mastodon.social/tags/datastreaming" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>datastreaming</span></a></p>
Block Rocker<p><span class="h-card" translate="no"><a href="https://mastodon.social/@warpstream" class="u-url mention" rel="nofollow noopener" target="_blank">@<span>warpstream</span></a></span> provides powerful and easy-to-use features for managing your <a href="https://mastodon.social/tags/ApacheKafka" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ApacheKafka</span></a> compatible topics in a WarpStream cluster. I made this 2-minute video to show you how it all works.</p><p><a href="https://mastodon.social/tags/dataengineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>dataengineering</span></a> <a href="https://mastodon.social/tags/datastreaming" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>datastreaming</span></a> <a href="https://mastodon.social/tags/kafka" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>kafka</span></a> <a href="https://mastodon.social/tags/streamingdata" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>streamingdata</span></a></p><p><a href="https://youtu.be/yn0dMWoGw4M" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">youtu.be/yn0dMWoGw4M</span><span class="invisible"></span></a></p>
Block Rocker<p>I had a great chat with 🐣 Simon Chaffetz at <span class="h-card" translate="no"><a href="https://mastodon.social/@warpstream" class="u-url mention" rel="nofollow noopener" target="_blank">@<span>warpstream</span></a></span> about <a href="https://mastodon.social/tags/ApacheKafka" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ApacheKafka</span></a> compatible BYOC clusters. We did a practical walkthrough using <a href="https://mastodon.social/tags/AWS" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AWS</span></a> and <a href="https://mastodon.social/tags/Kubernetes" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Kubernetes</span></a>. If you want to learn more about WarpStream and Bring Your Own Compute, then this is the video for you.</p><p><a href="https://mastodon.social/tags/dataengineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>dataengineering</span></a> <a href="https://mastodon.social/tags/streamingdata" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>streamingdata</span></a> <a href="https://mastodon.social/tags/datastreaming" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>datastreaming</span></a> <a href="https://mastodon.social/tags/kafka" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>kafka</span></a> </p><p><a href="https://youtu.be/Udt4dmpHCi8" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">youtu.be/Udt4dmpHCi8</span><span class="invisible"></span></a></p>
Alan Chen<p>If you're going to be in Austin, TX on Sept 19, and want to learn about event data streaming in Rust and WASM in a workshop setting, I have a number of discount seats to share.</p><p><a href="https://lu.ma/xdqcs436" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">lu.ma/xdqcs436</span><span class="invisible"></span></a> </p><p>DM me if you want to learn more, or are interested in a discount code.</p><p><a href="https://fosstodon.org/tags/rust" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>rust</span></a> <a href="https://fosstodon.org/tags/wasm" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>wasm</span></a> <a href="https://fosstodon.org/tags/datastreaming" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>datastreaming</span></a> <a href="https://fosstodon.org/tags/dataengineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>dataengineering</span></a></p>
Block Rocker<p>ShadowTraffic is a great tool for creating realistic sample data for just about anything. Recently, I've been using it in my content at <span class="h-card" translate="no"><a href="https://mastodon.social/@warpstream" class="u-url mention" rel="nofollow noopener" target="_blank">@<span>warpstream</span></a></span> to test out our software with other software in the ecosystem. It then occurred to me that I should get Michael Drogalis, the creator/founder, and create some content with him to show how it works and generate a couple of topics to illustrate it. </p><p><a href="https://mastodon.social/tags/apachekafka" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>apachekafka</span></a> <a href="https://mastodon.social/tags/kafka" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>kafka</span></a> <a href="https://mastodon.social/tags/dataengineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>dataengineering</span></a> <a href="https://mastodon.social/tags/datastreaming" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>datastreaming</span></a> <a href="https://mastodon.social/tags/streamingdata" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>streamingdata</span></a> <a href="https://mastodon.social/tags/testdata" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>testdata</span></a></p><p><a href="https://youtu.be/xHIk8p9Mp2Y" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">youtu.be/xHIk8p9Mp2Y</span><span class="invisible"></span></a></p>