Hey there Data Engineers. Want to stop putting out fires and start preventing them? Then it might be time to "shift left." By tackling quality, governance, and security from the get-go, you'll save time, money, and headaches.

If you want to learn more, follow the paywall bypassed link to my latest article. I hope some of you find this useful!

Time and again I see the same questions asked: "Why should I use dbt?" or "I don't understand what value dbt offers". So I thought I'd put together an article that touches on some of the benefits, as well as putting together a step through on setting up a new project (using DuckDB as the database), complete with associated GitHub repo for you to take a look at.

Having used dbt since early 2018, and with my partner being a dbt trainer, I hope that this article is useful for some of you. The link is paywall bypassed.

If you're a Data Engineer, before long you'll be asked to build a real-time pipeline.

In my latest article, I build a real-time pipeline using Kafka, Polars and Delta tables to demonstrate how these can work together. Everything is available to try yourself in the associated GitHub repo. So if you're curious, take a moment to check out this technical post.

How often do you build and edit Entity Relationship Diagrams? If the answer is ‘more often than I’d like’, and you’re fed up with tweaking your diagrams, take <5 minutes to read my latest article on building your diagrams with code. Track their changes in GitHub, have them build as part of your CI/CD pipeline, and even drop them into your dbt docs if you like.

This is a ‘friends and family’ link, so it’ll bypass the usual Medium paywall.

I’m not affiliated to the tool I’ve chosen in any way. Just like how it works.

Let me know yours thoughts!

I’ve written a series of Medium articles on creating a Data Pipeline from scratch, using Polars and DeltaTables. The first (linked) is an overview with link to the GitHub repository and each of the deeper dive articles. I then go into the next level of detail, walking through each component.

The articles are paywalled (it took time to build and document), but the link provided is the ‘family & friends’ link which bypasses the paywall for the Lemmy community.

I hope some of you may find this helpful.

A few years ago, if you'd mentioned Infrastructure-as-Code (IaC) to me, I would've given you a puzzled look. However I'm now on the bandwagon. And to help others understand how it can benefit them, I've pulled together a simple GitHub repo that showcases how Terraform can be used with Snowflake to manage users, roles, warehouses and databases.

The readme hopefully gives anyone who wants to give it a go the ability to step through and see results. I'm sharing this in the hopes that it is useful to some of you.

Hi all,

For those wanting a quick repo to use as a basis to get started, I’ve created jen-ai.

There are full instructions in the readme. Once running you can talk to it, and it will respond.

It’s basic, but a place to start.

The 2x OLED displays are still a work in progress

Hi all,

I'm hoping for some troubleshooting tips. I have a self-build split keyboard using Blackpills.

If I flash either side with 'SPLIT_KEYBOARD = no' in my rules.mk, then both sides work fine (but both working as the left side). If however I set 'SPLIT_KEYBOARD = yes', then both sides stop working.

It's driving me insane!

These are the relevant details of rules.mk:

MCU = STM32F401
BOOTLOADER = stm32-dfu
SPLIT_KEYBOARD = yes
SERIAL_DRIVER = usart

And config.h:

#pragma once
#define HAL_USE_SERIAL TRUE
#define SOFT_SERIAL_PIN A12
#define MASTER_LEFT
#define MATRIX_ROWS 6
#define MATRIX_COLS 6
#define SERIAL_USART_DRIVER SD1
#define SERIAL_USART_TX_PAL_MODE 7

And halconf.h:

#pragma once
#define SERIAL_USB_BUFFERS_SIZE 256
#include_next 

And mcuconf.h:

#pragma once
#include_next 
#undef STM32_SERIAL_USE_USART1
#define STM32_SERIAL_USE_USART1 TRUE

Please help! I've been at this for three days.

Hi all,

I'm hoping for some troubleshooting tips. I have a self-build split keyboard using Blackpills.

If I flash either side with 'SPLIT_KEYBOARD = no' in my rules.mk, then both sides work fine (but both working as the left side). If however I set 'SPLIT_KEYBOARD = yes', then both sides stop working.

It's driving me insane!

These are the relevant details of rules.mk:

MCU = STM32F401
BOOTLOADER = stm32-dfu
SPLIT_KEYBOARD = yes
SERIAL_DRIVER = usart

And config.h:

#pragma once
#define HAL_USE_SERIAL TRUE
#define SOFT_SERIAL_PIN A12
#define MASTER_LEFT
#define MATRIX_ROWS 6
#define MATRIX_COLS 6
#define SERIAL_USART_DRIVER SD1
#define SERIAL_USART_TX_PAL_MODE 7

And halconf.h:

#pragma once
#define SERIAL_USB_BUFFERS_SIZE 256
#include_next 

And mcuconf.h:

#pragma once
#include_next 
#undef STM32_SERIAL_USE_USART1
#define STM32_SERIAL_USE_USART1 TRUE

Please help! I've been at this for two days.

I created a 2d surface that I can perform a linear extrusion on, however the result it obviously a hard edge on the extrusion. I would love to be able to add a bezel - either rounded or at 45 degrees. Is there an easy way?