TechHub

0 posts0 participants0 posts today

R Tyler Croy 🦀Did you know you can purchase <a href="https://hacky.town/tags/deltalake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#deltalake</a> the definitive guide from reputable non-monopolistic book sellers?<a href="https://bookshop.org/p/books/delta-lake-the-definitive-guide-modern-data-lakehouse-architectures-with-data-lakes-denny-lee/21429337" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://bookshop.org/p/books/delta-lake-the-definitive-guide-modern-data-lakehouse-architectures-with-data-lakes-denny-lee/21429337</a>You can also pester some of the other authors and I next week in Mountain View, CA: <a href="https://lu.ma/okxq0bt1" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://lu.ma/okxq0bt1</a>

Knowledge Zone<a href="https://mstdn.social/tags/ITByte" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ITByte</a>: <a href="https://mstdn.social/tags/DeltaLake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DeltaLake</a> is an open-source <a href="https://mstdn.social/tags/Storage" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Storage</a> framework that enables building a <a href="https://mstdn.social/tags/Lakehouse" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Lakehouse</a> <a href="https://mstdn.social/tags/Architecture" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Architecture</a> with different compute engines.Delta Lake preserves the integrity of the original data without sacrificing the performance and agility required for real-time analytics, artificial intelligence (AI), and machine learning (ML) applications.<a href="https://knowledgezone.co.in/trends/explorer?topic=Delta-Lake" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://knowledgezone.co.in/trends/explorer?topic=Delta-Lake</a>

Markus SuhrJust caught up with the recent Delta Lake webinar, > Revolutionizing Delta Lake workflows on AWS Lambda with Polars, DuckDB, Daft & RustSome interesting hints there regarding lightweight processing of big-ish data. Easy to relate to any other framework instead of Lambda, e.g. <a href="https://gruene.social/tags/ApacheAirflow" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ApacheAirflow</a> tasks<a href="https://youtu.be/BR9oFD0QMAs" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://youtu.be/BR9oFD0QMAs</a><a href="https://gruene.social/tags/dataengineering" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#dataengineering</a> <a href="https://gruene.social/tags/datascience" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#datascience</a> <a href="https://gruene.social/tags/duckdb" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#duckdb</a> <a href="https://gruene.social/tags/daft" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#daft</a> <a href="https://gruene.social/tags/polars" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#polars</a> <a href="https://gruene.social/tags/pandas" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#pandas</a> <a href="https://gruene.social/tags/python" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#python</a> <a href="https://gruene.social/tags/spark" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#spark</a> <a href="https://gruene.social/tags/deltalake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#deltalake</a> <a href="https://gruene.social/tags/databricks" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#databricks</a> <a href="https://gruene.social/tags/airflow" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#airflow</a> <a href="https://gruene.social/tags/bigdata" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#bigdata</a> <a href="https://gruene.social/tags/smalldata" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#smalldata</a>

LavX NewsRevolutionizing Data Management: Mooncake-Labs Unveils pg_mooncake v0.1.0In a significant leap for data processing, Mooncake-Labs has launched pg_mooncake v0.1.0, introducing robust features that enhance PostgreSQL's capabilities. This release empowers users to seamlessly ...<a href="https://news.lavx.hu/article/revolutionizing-data-management-mooncake-labs-unveils-pg-mooncake-v0-1-0" rel="nofollow noopener noreferrer" target="_blank">https://news.lavx.hu/article/revolutionizing-data-management-mooncake-labs-unveils-pg-mooncake-v0-1-0</a><a href="https://mastodon.cloud/tags/news" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#news</a> <a href="https://mastodon.cloud/tags/tech" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#tech</a> <a href="https://mastodon.cloud/tags/PostgreSQL" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#PostgreSQL</a> <a href="https://mastodon.cloud/tags/DataManagement" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DataManagement</a> <a href="https://mastodon.cloud/tags/DeltaLake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DeltaLake</a>

LavX NewsRevolutionizing Data Management: Mooncake-Labs Unveils pg_mooncake v0.1.0In a significant leap for data processing, Mooncake-Labs has launched pg_mooncake v0.1.0, introducing robust features that enhance PostgreSQL's capabilities. This release empowers users to seamlessly ...<a href="https://news.lavx.hu/article/revolutionizing-data-management-mooncake-labs-unveils-pg-mooncake-v0-1-0" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://news.lavx.hu/article/revolutionizing-data-management-mooncake-labs-unveils-pg-mooncake-v0-1-0</a><a href="https://mastodon.social/tags/news" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#news</a> <a href="https://mastodon.social/tags/tech" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#tech</a> <a href="https://mastodon.social/tags/PostgreSQL" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#PostgreSQL</a> <a href="https://mastodon.social/tags/DataManagement" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DataManagement</a> <a href="https://mastodon.social/tags/DeltaLake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DeltaLake</a>

G.O.R.NDelta Lakeの楽観的排他制御：ACID特性の詳細と実現方法仕事で、Microsoft Fabric絡みのところを調べていました。基本、Fabricの中ではApache Sparkなどが動いていることは判っていたので、Delta LakeでACID特性が実現されているというのもなんとなくわかっていました。では、どのようにしてACID特性が実現されているのかというところには関心がありました。この辺に関して触れられているのは以下のQiita上の記事です。 <a href="https://qiita.com/toshimitsuk/items/5dac76dddfe921cb8a47" rel="nofollow noopener noreferrer" target="_blank">https://qiita.com/toshimitsuk/items/5dac76dddfe921cb8a47</a>そして、上記記事の次のところに着目しました。<blockquote>Delta Lakeのソースコード (<a href="https://github.com/delta-io/delta" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://github.com/delta-io/delta</a>) から、Delta Lakeの楽観的排他制御の仕組みを見ていきます。なお、ここではUpsertを対象に楽観的排他制御の仕組みを見ていきます。(再掲)</blockquote>なるほど、楽観的排他制御かと。では、楽観的排他制御とはなにかということです。悲観的排他制御これと対照的な概念が悲観的排他制御です。悲観的排他制御とは所謂ロックによって排他制御を実現します。こちらの方がよく知られているでしょう。いわゆるUpsertの場合において考えます。Upsertは排他制御がよくわかるところです、Upsertを仮に選択と条件分岐と更新と追加の組み合わせで考えます。<ol><li>対象キーでレコードが存在するかどうかを確認する</li><li>もし、レコードが存在すれば更新する</li><li>レコードが存在しなければ追加する</li></ol> これをそのまま実装してしまうと、1.と2.の間に割り込まれてしまうと、その間に重複するレコードが作成されてしまうリスクが発生します。結果的に、3.は正しく、Primary Keyなどの処理が実装されていればエラーになります。 Upsertであれば、悲観的排他制御であれば、1.のタイミングで対象となる表などにはロックが作られ、割り込み処理は一般的には待たされます。結果として、Upsertは完遂できます。仮に、待たされた方が自動連番などでキーを発生させていれば、その後により後のキーが発行されて正常に終了するでしょう。楽観的排他制御 では、これが楽観的排他制御ではどうなるでしょうか? 楽観的排他制御ではロックではなく、更新の検出により処理します。つまり、処理する前に、割り込まれたことを検出して処理をします。つまり、なぜ楽観的なのかというと、滅多に抵触する処理は行われないはずという仮定があります。このあたりの詳しい話については以下の記事も参考にできます。 <a href="https://qiita.com/NagaokaKenichi/items/73040df85b7bd4e9ecfc" rel="nofollow noopener noreferrer" target="_blank">https://qiita.com/NagaokaKenichi/items/73040df85b7bd4e9ecfc</a><ol><li>検証: 最初にレコードの現在の状態を取得し、その状態を記録します</li><li>更新の準備: データを更新する準備をします</li><li>更新の試行: 更新を試みますが、その際に他のトランザクションが同じレコードを変更していないかを確認します</li><li>競合の対処: 他のトランザクションが変更していた場合は、更新を再試行するか、エラーを返します</li></ol>パフォーマンスと課題さて、悲観的排他制御と楽観的排他制御の差はどこにあるのか? 一つのポイントはパフォーマンスです。悲観的排他制御はロックを行うため、そこがパフォーマンス上のボトルネックになります。では、楽観的排他制御は銀の弾丸か? そんなことはありません。銀の弾丸はどこにも存在しないのです。楽観的排他制御の弱点は、更新の検出にあります。これが崩れればすべてがお釈迦です。そして、楽観的排他制御のポイントは、楽観的、競合することはそうそう起きないだろうにあります。競合がかなりの頻度で起きるとすればどうでしょうか? ほとんどの場合で、何らかの回避動作が必要になります。更に、その回避動作は正しいのかの検証も極めて重要です。結論から言うと、楽観的排他制御は難易度が高いです。その意味ではロックして、競合処理を起こさないようにする悲観的排他制御の方がシステム的な難易度は低いです。パフォーマンスの劣化は大きな課題ですが。<a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://tech.grayrecord.com/tag/delta-lake/" target="_blank">#DeltaLake</a> <a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://tech.grayrecord.com/tag/%e6%8e%92%e4%bb%96%e5%88%b6%e5%be%a1/" target="_blank">#排他制御</a>

Alex MercedNew Blog on Table Format Interoperability:<a href="https://medium.com/data-engineering-with-dremio/when-to-use-apache-xtable-or-delta-lake-uniform-for-data-lakehouse-interoperability-d89b120e2163" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://medium.com/data-engineering-with-dremio/when-to-use-apache-xtable-or-delta-lake-uniform-for-data-lakehouse-interoperability-d89b120e2163</a><a href="https://mastodon.social/tags/dataengineering" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#dataengineering</a> <a href="https://mastodon.social/tags/apacheiceberg" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#apacheiceberg</a> <a href="https://mastodon.social/tags/deltalake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#deltalake</a> <a href="https://mastodon.social/tags/apachehudi" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#apachehudi</a> <a href="https://mastodon.social/tags/datalakehouse" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#datalakehouse</a>

R Tyler Croy 🦀I hope you'll be able to join the <a href="https://hacky.town/tags/deltalake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#deltalake</a> AMA today at 9am PST with the authors of Delta Lake The Definitive Guide 🤗 <a href="https://www.linkedin.com/events/asktheauthors7257759127987261441/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://www.linkedin.com/events/asktheauthors7257759127987261441/</a>

R Tyler Croy 🦀I am so so excited to have a physical copy of <a href="https://hacky.town/tags/deltalake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#deltalake</a> "The Definitive Guide"

R Tyler Croy 🦀I had some talks earlier this year about <a href="https://hacky.town/tags/deltalake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#deltalake</a> and <a href="https://hacky.town/tags/rustlang" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#rustlang</a> I also did so with one hand...because I broke my wrist at the beginning of the conference 🤦 <a href="https://brokenco.de/2024/10/17/data-ai-summit-videos.html" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://brokenco.de/2024/10/17/data-ai-summit-videos.html</a>

R Tyler Croy 🦀If you use <a href="https://hacky.town/tags/deltalake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#deltalake</a> from Python or Rust, I sure would appreciate it if your company sponsored my work!<a href="https://github.com/sponsors/rtyler" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://github.com/sponsors/rtyler</a>

R Tyler Croy 🦀If you use <a href="https://hacky.town/tags/deltalake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#deltalake</a> for <a href="https://hacky.town/tags/python" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#python</a>, you can now sponsor one of it's main contributors Ion! <a href="https://github.com/ion-elgreco" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://github.com/ion-elgreco</a>

Delta LakeWe are excited to announce the release of python-v0.19.0! 🎉 Highlights include: 🌟 CDF support in write_deltalake, delete, and merge operation 🌟 Expired logs cleanup during post-commit. Can be disabled with 𝚍𝚎𝚕𝚝𝚊.𝚎𝚗𝚊𝚋𝚕𝚎𝙴𝚡𝚙𝚒𝚛𝚎𝚍𝙻𝚘𝚐𝙲𝚕𝚎𝚊𝚗𝚞𝚙 = 𝚏𝚊𝚕𝚜𝚎 🌟 Improved MERGE performance by using predicate non-partition columns min/max for prefiltering 🌟 𝙰𝙳𝙳 𝚌𝚘𝚕𝚞𝚖𝚗 option 🌟 Speed up log parsing🔗 View the release notes: <a href="https://github.com/delta-io/delta-rs/releases/tag/python-v0.19.0" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://github.com/delta-io/delta-rs/releases/tag/python-v0.19.0</a><a href="https://social.lfx.dev/tags/deltalake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#deltalake</a> <a href="https://social.lfx.dev/tags/rust" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#rust</a> <a href="https://social.lfx.dev/tags/linuxfoundation" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#linuxfoundation</a> <a href="https://social.lfx.dev/tags/opensource" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#opensource</a>

Jörn Franke<a href="https://mastodon.online/tags/apache" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#apache</a> <a href="https://mastodon.online/tags/xtable" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#xtable</a> seamlessly interoperate cross-table between apache <a href="https://mastodon.online/tags/hudi" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hudi</a>, <a href="https://mastodon.online/tags/deltalake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#deltalake</a>, and apache <a href="https://mastodon.online/tags/iceberg" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#iceberg</a> <a href="https://xtable.apache.org/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://xtable.apache.org/</a>

R Tyler Croy 🦀The <a href="https://hacky.town/tags/deltalake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#deltalake</a> Python and Rust bindings have change data capture (CDC) read and write support in the works right now.*Very* exciting stuff happening between now and Data and AI Summit in June!

R Tyler Croy 🦀Finally hitting a groove with this <a href="https://hacky.town/tags/deltalake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#deltalake</a> code at ... almost 23:00 local time 🤢

Delta Lake⭐ Join us for the next <a href="https://social.lfx.dev/tags/DeltaLake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DeltaLake</a> Community Meetup on Wednesday, May 22nd at 9AM PST / 12PM EST.In this virtual event, we will cover the latest features and improvements in Delta Lake 3.2, walk through the Delta Lake + Pandas integration, discuss what's next for Delta Lake, and more. 🦀 Bring your <a href="https://social.lfx.dev/tags/DeltaLake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DeltaLake</a> questions and get instant feedback from core contributors and maintainers of the project.Register here 👉 <a href="https://us06web.zoom.us/webinar/register/WN_9yqi7z2wSuCtVqqcuAeA8g#/registration" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://us06web.zoom.us/webinar/register/WN_9yqi7z2wSuCtVqqcuAeA8g#/registration</a><a href="https://social.lfx.dev/tags/linuxfoundation" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#linuxfoundation</a> <a href="https://social.lfx.dev/tags/oss" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#oss</a>

Delta LakeWe are excited to announce the first release candidate for <a href="https://social.lfx.dev/tags/DeltaLake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DeltaLake</a> 3.2.0! 🚀Some highlights include: ✅ Support for Liquid clustering to reduce write amplification using incremental clustering. ✅ Support for Type Widening (preview) to allow users to change the type of columns without having to rewrite data. ✅ Support for Apache Hudi (preview) in Delta UniForm tables. ✅ Support for Vacuum Inventory.Please see the full release notes for more info: <a href="https://github.com/delta-io/delta/releases/tag/v3.2.0rc1" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://github.com/delta-io/delta/releases/tag/v3.2.0rc1</a><a href="https://social.lfx.dev/tags/oss" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#oss</a> <a href="https://social.lfx.dev/tags/opensource" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#opensource</a>

R Tyler Croy 🦀I had fun sharing the latest and greatest in the <a href="https://hacky.town/tags/deltalake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#deltalake</a> <a href="https://hacky.town/tags/rustlang" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#rustlang</a> project.Our next release 0.17 will include some really important enhancements which will go live next week!Learn more here: <a href="https://youtu.be/300wnXIwVP0?t=327" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://youtu.be/300wnXIwVP0?t=327</a>

Recent searches

Search options

Administered by:

Server stats:

#deltalake