Data-Engineering on Lough on Data

Data-Engineering on Lough on Data https://loughondata.com/tags/data-engineering/ Recent content in Data-Engineering on Lough on Data Hugo -- gohugo.io en connor@loughondata.com (Connor Lough) connor@loughondata.com (Connor Lough) © 2026 Connor Lough Sat, 02 May 2026 00:00:00 +0000 Local Data Stack https://loughondata.com/posts/local-data-stack-build/ Sat, 02 May 2026 00:00:00 +0000 connor@loughondata.com (Connor Lough) https://loughondata.com/posts/local-data-stack-build/ The follow-up to Part 1. I built the stack — a small dlt pipeline that watches a hand-picked list of outdoor, geospatial, and forestry companies for new roles, writes Parquet to Cloudflare R2, and surfaces everything in a MotherDuck Dive I check on my phone. Here’s how it works, what changed from the original plan, and what I deliberately left naive. Databox https://loughondata.com/projects/databox/ Tue, 28 Apr 2026 00:00:00 +0000 connor@loughondata.com (Connor Lough) https://loughondata.com/projects/databox/ A single-operator data platform that ingests eBird, NOAA, and USGS into one queryable warehouse — built to answer whether species distributions shift with same-day weather and streamflow anomalies. Local Data Stack Planning https://loughondata.com/posts/local-data-stack-planning/ Tue, 28 Apr 2026 00:00:00 +0000 connor@loughondata.com (Connor Lough) https://loughondata.com/posts/local-data-stack-planning/ On a dog-walk conversation with Claude that surfaced a role I would have otherwise missed — and the small dlt → S3 → MotherDuck pipeline I’m building so it doesn’t happen again. Never Make Another dbt model .yml https://loughondata.com/posts/never-make-another-yml/ Sat, 10 Feb 2024 00:00:00 +0000 connor@loughondata.com (Connor Lough) https://loughondata.com/posts/never-make-another-yml/ A GitHub Action that runs dbt-osmosis + dbt2looker on every push, then commits the regenerated .yml and .lookml files back to your repo. dlt Demo: Bird Hotspot Finder, pt. II https://loughondata.com/posts/bird-finder-via-dlt-ii/ Thu, 15 Jun 2023 00:00:00 +0000 connor@loughondata.com (Connor Lough) https://loughondata.com/posts/bird-finder-via-dlt-ii/ Extending the dlt pipeline: stage load packages to S3, persist schema changes back to DuckDB, and ping Slack on schema drift. dlt Demo: Bird Hotspot Finder, pt. I https://loughondata.com/posts/bird-finder-via-dlt-i/ Thu, 01 Jun 2023 00:00:00 +0000 connor@loughondata.com (Connor Lough) https://loughondata.com/posts/bird-finder-via-dlt-i/ Loading eBird API data into DuckDB with dlt, then querying it in DBeaver. A first look at dlt as a Meltano/Airbyte alternative. Creating a Mermaid Diagram for Your dbt Project Using graph.gpickle https://loughondata.com/posts/dbt-and-mermaid-diagramming/ Sat, 15 Apr 2023 00:00:00 +0000 connor@loughondata.com (Connor Lough) https://loughondata.com/posts/dbt-and-mermaid-diagramming/ Turn a dbt project’s compiled graph.gpickle into a Mermaid diagram so you can actually see what your models depend on. Creating a Data Pipeline Between PostgreSQL and Snowflake with Meltano https://loughondata.com/posts/meltano-getting-started/ Wed, 05 Apr 2023 00:00:00 +0000 connor@loughondata.com (Connor Lough) https://loughondata.com/posts/meltano-getting-started/ Standing up a Meltano pipeline from a PostgreSQL source to a Snowflake destination — init, plugins, env config, run.