Sarah Lea<p>Why normalize databases?<br />Yesterday, my tutoring student asked me why databases need to be normalized at all. She said: “Wouldn’t it be easier to just have one big table with all the information?”</p><p>It’s a common first question when learning about relational databases.<br />At first, one big table (e.g. customer name, order date, product name, price) seems easiest.</p><p>I told her:<br />:blobcoffee: Because that quickly leads to data redundancy, anomalies, and integrity issues when inserting, updating, or deleting records.<br />:blobcoffee: Normalization means structuring data into separate, related tables, so that each fact is stored only once. This reduces redundancy & preserves consistency.</p><p><a href="https://techhub.social/tags/databases" class="mention hashtag" rel="tag">#<span>databases</span></a> <a href="https://techhub.social/tags/dataengineering" class="mention hashtag" rel="tag">#<span>dataengineering</span></a> <a href="https://techhub.social/tags/datascience" class="mention hashtag" rel="tag">#<span>datascience</span></a> <a href="https://techhub.social/tags/datascientist" class="mention hashtag" rel="tag">#<span>datascientist</span></a> <a href="https://techhub.social/tags/dataanalysis" class="mention hashtag" rel="tag">#<span>dataanalysis</span></a> <a href="https://techhub.social/tags/dataanalyst" class="mention hashtag" rel="tag">#<span>dataanalyst</span></a> <a href="https://techhub.social/tags/sql" class="mention hashtag" rel="tag">#<span>sql</span></a> <a href="https://techhub.social/tags/data" class="mention hashtag" rel="tag">#<span>data</span></a></p>
techhub.social is one of the many independent Mastodon servers you can use to participate in the fediverse.
A hub primarily for passionate technologists, but everyone is welcome
Administered by:
Server stats:
4.6Kactive users
techhub.social: About · Status · Profiles directory · Privacy policy
Mastodon: About · Get the app · Keyboard shortcuts · View source code · v4.4.2
#DataEngineering
17 posts · 15 participants · 1 post today
The Data Channel<p>Data Engineering Tip Day 21: Make Testing a First-Class Citizen</p><p>🔹 Tip: Write unit tests, data tests, and pipeline tests using tools like pytest, dbt tests, or Great Expectations.</p><p>🔸 Why?: Testing ensures correctness in production and builds trust. Test before you ingest, transform, or publish data.</p><p><a href="https://mastodon.social/tags/theDataChannel" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>theDataChannel</span></a> <a href="https://mastodon.social/tags/dataengineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>dataengineering</span></a></p>
Glasskonnect<p>🚀 Kickstart Your Career in Data Engineering!</p><p>Join our Data Engineering Course at GlassKonnect and build a strong foundation with real-time training and expert mentorship. Gain the skills that top companies demand today!</p><p>📞 Call us: +91 86884 53557<br>📧 Mail: info@glasskonnect.com<br>🌐 Visit: www.glasskonnect.com</p><p>📍Limited Seats | 📆 Weekend Batches Available<br><a href="https://mastodon.social/tags/DataEngineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DataEngineering</span></a> <a href="https://mastodon.social/tags/GlassKonnect" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>GlassKonnect</span></a> <a href="https://mastodon.social/tags/TechCareers" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>TechCareers</span></a> <a href="https://mastodon.social/tags/ITTraining" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ITTraining</span></a> meSupport <a href="https://mastodon.social/tags/PlacementAssistance" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>PlacementAssistance</span></a></p>
The Data Channel<p>Data Engineering Tip of the Day: Day 20: Choose the Right File Format</p><p>🔹 Tip: Use columnar formats like Parquet, ORC, or Delta for analytics; use Avro/JSON for row-wise or streaming.</p><p>🔸 Why?: File format choice impacts compression, speed, and compatibility. Match the format to the use case.</p><p><a href="https://mastodon.social/tags/theDataChannel" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>theDataChannel</span></a> <a href="https://mastodon.social/tags/dataengineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>dataengineering</span></a></p>
pipTrends<p>This week's newsletter will be out tomorrow. Interesting stuff by <span class="h-card" translate="no"><a href="https://fosstodon.org/@django" class="u-url mention" rel="nofollow noopener" target="_blank">@<span>django</span></a></span> <span class="h-card" translate="no"><a href="https://mas.to/@s_gruppetta" class="u-url mention" rel="nofollow noopener" target="_blank">@<span>s_gruppetta</span></a></span> Hugo van Kemenade <span class="h-card" translate="no"><a href="https://hachyderm.io/@mitsuhiko" class="u-url mention" rel="nofollow noopener" target="_blank">@<span>mitsuhiko</span></a></span> & Jordi Esteve Sorribas covered</p><p><a href="https://newsletter.piptrends.com/p/djangos-ecosystem-python-backstage" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">newsletter.piptrends.com/p/dja</span><span class="invisible">ngos-ecosystem-python-backstage</span></a></p><p><a href="https://mastodon.social/tags/python" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>python</span></a> <a href="https://mastodon.social/tags/Programming" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Programming</span></a> <a href="https://mastodon.social/tags/PythonProgramming" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>PythonProgramming</span></a> <a href="https://mastodon.social/tags/ai" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ai</span></a> <a href="https://mastodon.social/tags/ml" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ml</span></a> <a href="https://mastodon.social/tags/MachineLearning" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>MachineLearning</span></a> <a href="https://mastodon.social/tags/SoftwareDevelopment" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>SoftwareDevelopment</span></a> <a href="https://mastodon.social/tags/WebDevelopment" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>WebDevelopment</span></a> <a href="https://mastodon.social/tags/TechNews" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>TechNews</span></a> <a href="https://mastodon.social/tags/OpenSource" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>OpenSource</span></a> <a href="https://mastodon.social/tags/DataEngineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DataEngineering</span></a></p>
pipTrends<p>If you think you know a lot about f-strings, try this quiz made by <span class="h-card" translate="no"><a href="https://hachyderm.io/@mitsuhiko" class="u-url mention" rel="nofollow noopener" target="_blank">@<span>mitsuhiko</span></a></span> I scored 11 and learned that you can do many cool things with f-strings. I hope you have fun & learning too!</p><p><a href="https://fstrings.wtf/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">fstrings.wtf/</span><span class="invisible"></span></a></p><p><a href="https://mastodon.social/tags/python" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>python</span></a> <a href="https://mastodon.social/tags/Programming" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Programming</span></a> <a href="https://mastodon.social/tags/PythonProgramming" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>PythonProgramming</span></a> <a href="https://mastodon.social/tags/ai" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ai</span></a> <a href="https://mastodon.social/tags/ml" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ml</span></a> <a href="https://mastodon.social/tags/MachineLearning" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>MachineLearning</span></a> <a href="https://mastodon.social/tags/SoftwareDevelopment" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>SoftwareDevelopment</span></a> <a href="https://mastodon.social/tags/WebDevelopment" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>WebDevelopment</span></a> <a href="https://mastodon.social/tags/TechNews" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>TechNews</span></a> <a href="https://mastodon.social/tags/OpenSource" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>OpenSource</span></a> <a href="https://mastodon.social/tags/DataEngineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DataEngineering</span></a></p>
pipTrends<p>We all know that the Python interpreter changes our code into an intermediate form called bytecode. With the dis module, you can see what this bytecode looks like. In this article, <span class="h-card" translate="no"><a href="https://mas.to/@s_gruppetta" class="u-url mention" rel="nofollow noopener" target="_blank">@<span>s_gruppetta</span></a></span> explained what happens inside when we run a Python program.</p><p><a href="https://www.thepythoncodingstack.com/p/python-disassembling-bytecode-dis-module" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://www.</span><span class="ellipsis">thepythoncodingstack.com/p/pyt</span><span class="invisible">hon-disassembling-bytecode-dis-module</span></a></p><p><a href="https://mastodon.social/tags/python" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>python</span></a> <a href="https://mastodon.social/tags/Programming" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Programming</span></a> <a href="https://mastodon.social/tags/PythonProgramming" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>PythonProgramming</span></a> <a href="https://mastodon.social/tags/ai" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ai</span></a> <a href="https://mastodon.social/tags/ml" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ml</span></a> <a href="https://mastodon.social/tags/MachineLearning" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>MachineLearning</span></a> <a href="https://mastodon.social/tags/SoftwareDevelopment" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>SoftwareDevelopment</span></a> <a href="https://mastodon.social/tags/WebDevelopment" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>WebDevelopment</span></a> <a href="https://mastodon.social/tags/TechNews" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>TechNews</span></a> <a href="https://mastodon.social/tags/OpenSource" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>OpenSource</span></a> <a href="https://mastodon.social/tags/DataEngineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DataEngineering</span></a></p>
Thewatch<p>What’s a data engineer’s favorite way to party?</p><p>:computerfairies: :calculator: :toot: :bongoCat: </p><p>Join tables… then group together by friends! </p><p><a href="https://mastodon.social/tags/data" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>data</span></a> <a href="https://mastodon.social/tags/dataengineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>dataengineering</span></a> <a href="https://mastodon.social/tags/sql" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>sql</span></a> <a href="https://mastodon.social/tags/meme" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>meme</span></a> <a href="https://mastodon.social/tags/pipeline" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>pipeline</span></a></p>
The Data Channel<p>Data engineering Tip of the Day: Day 19: Use CDC Over Full Loads When Possible</p><p>🔹 Tip: Implement Change Data Capture (CDC) instead of full refreshes, especially on large transactional tables.</p><p>🔸 Why?: Saves compute, improves freshness. Use tools like Debezium, Fivetran, Oracle GoldenGate, or native database CDC features.</p><p><a href="https://mastodon.social/tags/theDataChannel" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>theDataChannel</span></a> <a href="https://mastodon.social/tags/dataEngineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>dataEngineering</span></a></p>
Browsejobs<p>🚀 Data Engineers are the new rockstars of Indian tech—and for good reason! From powering AI to building the backbone of analytics, their role is more critical than ever. If you're exploring high-growth tech careers in India, this one's for you.</p><p>🔗 Read the full blog: <a href="https://dev.to/browsejobs/data-engineers-are-the-new-rockstars-of-indian-tech-heres-why-1dfl" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">dev.to/browsejobs/data-enginee</span><span class="invisible">rs-are-the-new-rockstars-of-indian-tech-heres-why-1dfl</span></a></p><p><a href="https://mastodon.social/tags/DataEngineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DataEngineering</span></a> <a href="https://mastodon.social/tags/IndianTech" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>IndianTech</span></a> <a href="https://mastodon.social/tags/TechCareers" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>TechCareers</span></a> <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AI</span></a> <a href="https://mastodon.social/tags/BrowseJobs" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>BrowseJobs</span></a> <a href="https://mastodon.social/tags/UpskillIndia" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>UpskillIndia</span></a> <a href="https://mastodon.social/tags/Python" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Python</span></a> <a href="https://mastodon.social/tags/BigData" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>BigData</span></a> <a href="https://mastodon.social/tags/CareerGrowth" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CareerGrowth</span></a> <a href="https://mastodon.social/tags/DataJobs" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DataJobs</span></a></p>
pipTrends<p>A programming language or framework becomes popular when it offers numerous helpful tools and resources. <span class="h-card" translate="no"><a href="https://fosstodon.org/@django" class="u-url mention" rel="nofollow noopener" target="_blank">@<span>django</span></a></span> is one of the great examples of it. The Django Software Foundation has put together a useful list with information, including packages for debugging, API development and more.</p><p><a href="https://www.djangoproject.com/community/ecosystem/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://www.</span><span class="ellipsis">djangoproject.com/community/ec</span><span class="invisible">osystem/</span></a></p><p><a href="https://mastodon.social/tags/python" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>python</span></a> <a href="https://mastodon.social/tags/Programming" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Programming</span></a> <a href="https://mastodon.social/tags/PythonProgramming" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>PythonProgramming</span></a> <a href="https://mastodon.social/tags/ai" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ai</span></a> <a href="https://mastodon.social/tags/ml" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ml</span></a> <a href="https://mastodon.social/tags/MachineLearning" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>MachineLearning</span></a> <a href="https://mastodon.social/tags/SoftwareDevelopment" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>SoftwareDevelopment</span></a> <a href="https://mastodon.social/tags/WebDevelopment" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>WebDevelopment</span></a> <a href="https://mastodon.social/tags/TechNews" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>TechNews</span></a> <a href="https://mastodon.social/tags/OpenSource" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>OpenSource</span></a> <a href="https://mastodon.social/tags/DataEngineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DataEngineering</span></a></p>
The Data Channel<p>Data Dose of the Day: Day 18</p><p>🔹 Tip: Keep compute and storage in the same region or zone to minimize latency and egress costs.</p><p>🔸 Why?: Cloud costs can spike with inefficient architecture. Know where your data and jobs physically reside.</p><p><a href="https://mastodon.social/tags/theDataChannel" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>theDataChannel</span></a> <a href="https://mastodon.social/tags/DataEngineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DataEngineering</span></a></p>
TuxAcademy<p>Today's Poll...</p><p>What is a DDL statement in SQL? </p><p><a href="https://mastodon.social/tags/SQL" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>SQL</span></a> <a href="https://mastodon.social/tags/Database" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Database</span></a> <a href="https://mastodon.social/tags/TechQuiz" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>TechQuiz</span></a> <a href="https://mastodon.social/tags/LearnSQL" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>LearnSQL</span></a> <a href="https://mastodon.social/tags/TuxAcademy" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>TuxAcademy</span></a> <a href="https://mastodon.social/tags/ProgrammingFacts" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ProgrammingFacts</span></a> <a href="https://mastodon.social/tags/InterviewPrep" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>InterviewPrep</span></a> <a href="https://mastodon.social/tags/DataEngineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DataEngineering</span></a> <a href="https://mastodon.social/tags/MySQL" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>MySQL</span></a></p>
Thewatch<p><a href="https://mastodon.social/tags/data" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>data</span></a> <a href="https://mastodon.social/tags/dataengineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>dataengineering</span></a> <a href="https://mastodon.social/tags/datascience" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>datascience</span></a> <a href="https://mastodon.social/tags/etl" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>etl</span></a> <a href="https://mastodon.social/tags/DataPipe" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DataPipe</span></a> <a href="https://mastodon.social/tags/sql" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>sql</span></a> <a href="https://mastodon.social/tags/sqlserver" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>sqlserver</span></a> <a href="https://mastodon.social/tags/meme" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>meme</span></a> <a href="https://mastodon.social/tags/memes" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>memes</span></a></p>
Thewatch<p><a href="https://mastodon.social/tags/data" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>data</span></a> <a href="https://mastodon.social/tags/dataengineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>dataengineering</span></a> <a href="https://mastodon.social/tags/meme" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>meme</span></a> <a href="https://mastodon.social/tags/sql" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>sql</span></a> <a href="https://mastodon.social/tags/snowflake" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>snowflake</span></a> <a href="https://mastodon.social/tags/DataPipe" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DataPipe</span></a></p>
Posit<p>What makes tools truly useful? </p><p>Episode 2 of <a href="https://fosstodon.org/tags/TheTestSet" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>TheTestSet</span></a> features Wes McKinney (Part 1of 2!) sharing his experience building Pandas & Arrow, plus his surprising past in speedrun communities.</p><p>Tune in for his story at thetestset.co, on Spotify, or Apple Podcasts</p><p><a href="https://fosstodon.org/tags/DataStack" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DataStack</span></a> <a href="https://fosstodon.org/tags/DataEngineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DataEngineering</span></a> <a href="https://fosstodon.org/tags/OpenSource" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>OpenSource</span></a> <a href="https://fosstodon.org/tags/Podcast" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Podcast</span></a> <a href="https://fosstodon.org/tags/Python" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Python</span></a></p>
Thewatch<p>I asked my ETL job how life was going.<br><a href="https://mastodon.social/tags/etl" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>etl</span></a> <a href="https://mastodon.social/tags/dataengineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>dataengineering</span></a> <a href="https://mastodon.social/tags/data" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>data</span></a> <a href="https://mastodon.social/tags/sql" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>sql</span></a> <a href="https://mastodon.social/tags/meme" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>meme</span></a></p>
Andrea D'Ambrosio<p>Data engineering vs AI engineering career sustainability analysis:</p><p>Infrastructure skills have longer technology cycle persistence than application-layer skills.</p><p>Open source data tools (PostgreSQL, Kafka) vs proprietary AI frameworks demonstrates knowledge durability advantage.</p><p><a href="https://mastodon.social/tags/OpenSource" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>OpenSource</span></a> <a href="https://mastodon.social/tags/DataEngineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DataEngineering</span></a></p>
The Data Channel<p>Data Engineering Tip of the Day: Day 16</p><p>🔹 Tip: Integrate data lineage tools like OpenLineage, Marquez, Azure Purview, or Databricks Unity Catalog.</p><p>🔸 Why?: Lineage helps trace errors, manage impact of changes, and improves governance. Essential for compliance and debugging.</p><p><a href="https://mastodon.social/tags/datadoseoftheday" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>datadoseoftheday</span></a> <a href="https://mastodon.social/tags/DataEngineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DataEngineering</span></a></p>
The Data Channel<p>Data Engineering Tip: Day 15</p><p>🔹 Tip: Design your pipelines with backfill support in mind—parameterized by date/time or partition.</p><p>🔸 Why?: Reprocessing historical data becomes painless. Avoid hardcoded date filters and use config-driven ranges.</p><p><a href="https://mastodon.social/tags/DataEngineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DataEngineering</span></a> <a href="https://mastodon.social/tags/datadoseoftheday" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>datadoseoftheday</span></a> <a href="https://mastodon.social/tags/theDataChannel" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>theDataChannel</span></a></p>
TrendingLive feeds
Mastodon is the best way to keep up with what's happening.
Follow anyone across the fediverse and see it all in chronological order. No algorithms, ads, or clickbait in sight.
Create accountLoginDrag & drop to upload