-
State of Data Engineering Q3 2024
Here is this quarter’s state of data engineering newsletter. There is only a little chat about AI this time, and a focus on Open Table Formats, the Apache Iceberg Rest Spec, Open Table Format updates, and new updates in the Amazon Data Engineering ecosystem. Prompt Engineering – Meta Analysis Whitepaper One of my favorite AI…
-
State of Data Engineering 2024 Q2
Data Engineering and AIChip Huyen, who came out of Stanford and is active in the AI space recently wrote an article on what she learned by looking at the 900 most popular open source AI tools. https://huyenchip.com/2024/03/14/ai-oss.html Image Credit: huyenchip’s blog In data engineering, one of our primary usages of AI is really just prompt engineering. Use Case 1: Data Migration…
-
State of Data Engineering 2024 Q1
The current state of data engineering offers a plethora of options in the market, which can be challenging when selecting the right tool We are approaching a period where the traditional boundaries between between databases, datalakes, and data warehouses are overlapping. As always, it is important to think about what is the business case, then…
-
State of Data Engineering 2023 Q3
As we roll towards the end of the year data engineering as expected does have some changes, but now everyone wants to see how Generative AI intersects with everything. The fits are not completely natural, as Generative AI like Chat GPT is more NLP type systems, but there are a few interesting cases to keep…
-
State of Data Engineering 2023 Q2
When looking at data engineering for your projects, it is important to think about market segmentation. In particular, you might be able to think about it in four segments Small Data – This refers to scenarios where companies have data problems (organization, modeling, normalization, etc), but don’t necessarily generate a ton of data. When you…
-
Data Engineering Low Code Tools
In the data engineering space we have seen quite a few low code and no code tools pass through our radar. Low code tools have their own nuances as you will get to operationalize quicker, but the minute you need to customize something outside of the toolbox, you may run into problems. That’s when we…
-
Software Estimations Using Reference Class Forecasting
18 years ago I’m sitting in my cubicle doing Java programming, and my tech lead comes up to me to chat about my next project. We discuss the details, and then she asks me the dreaded questions programmers fear which is “how long will it take?”. I stumble with some guestimate based off my limited…
-
West Coast Trail – The 75km/48 mile death hike
Author Note: This trip was taken in 2021, but updated in 2023 with updated details. I’m not really sure where I get these crazy ideas, but a friend and I booked the West Coast Trail. It is this multi day thru hike in the west coast of Vancouver Island, which is accessible via ferry. Unfortunately in 2020 the hike was canceled,…