When looking at data engineering for your projects, it is important to think about market segmentation. In particular, you might be able to think about it in four segments

• Small Data
• Medium Data
• Big Data
• Lots and Lots of Data
  

Small Data – This refers to scenarios where companies have data problems (organization, modeling, normalization, etc), but don’t necessarily generate a ton of data. When you don’t have a lot of data, different tool sets are in use ranging from low code tools to simpler storage mechanisms like SQL databases.

 
Low Code Tools 

The market is saturated with low code tools, with an estimated 80-100 products available. Whether low code tools work for you depends on your use case. If your teams lack a strong engineering capacity, it makes sense to use a tool to help accomplish ETL tasks.

However, problems arise when customers need to do something outside the scope of the tool.

Medium Data– This refers to customers who have more data, making it sensible to leverage more powerful tools like Spark. There are several ways to solve the problem with data lakes, data warehouses, ETL, or reverse ETL.

Big Data – This is similar to medium data, but introduces the concepts of incremental ETL (aka transactional data lakes or lake houses). Customers in this space tend to have data in the hundreds gigabytes to terabytes.

Transactional data lakes are essential because incremental ETL is challenging. For example, consider an Uber ride to the airport that costs $30. Later, you give a $5 tip, and now your trip costs $35. In a traditional database, you can run some ETL to update the script. However, Uber has tons of transactions worldwide, and they need a different way of dealing with the problem.

Introducing transactional data lakes requires more operational overhead, which should be taken into consideration.

Lots and Lots of Data – Customers in this space generate terabytes or petabytes of data a day. For example, Walmart creates 10 pb of data (!) a day.

https://medium.com/walmartglobaltech/lakehouse-at-fortune-1-scale-480bcb10391b

When customers are in this space, transactional data lakes with Apache Hudi, Apache Iceberg, and Databricks Deltalake are the main tools used.

Conclusion

The data space is large and crowded. With the small and lots of data sizes, the market segment is clear. However, the mid-market data space will probably take some time for winners to emerge.

Easter Island

Before going on any travel trips, my travel style is usually to do as much research as possible.  My general methodology is to ask friends for advice, read guide books like Lonely Planet, look up travel itineraries on Reddit, and try to schedule zoom calls with travel writers.

The last idea of scheduling zoom calls with travel writers was actually a new idea that has been rather successful.  When we were planning Portugal, we had to make some major decisions on travel routing, and the guide books didn’t provide clear guidance one way or another.  One day I was listening to a Rick Steve’s podcast about Portugal, and a local guide Cristina Duarte seemed rather knowledgeable.  I decided to cold e-mail her to ask if she would be open to doing a one hour paid zoom session.

She actually responded pretty quickly via email and then we scheduled a session on zoom in a couple days.  With zoom we had a google sheets up with a draft itinerary and a google map to ask questions about locations.  Cristina was able to guide us on some major decision points as well as a lot of tips about the country.

This model was so successful in helping plan the Portugal trip that we did the same thing again with another travel writer Mark Johanson for Chile.

As we talked to Mark, he guided us on selecting a few regions, and for Easter Island said you have to book a guide to see the sights so you can’t see things on your own.   Post pandemic, government regulations have changed (I think due to the better) to help protect the moais from tourists.

Oddly enough, I think my first exposure to Easter Island was when I was 8 years old playing a Nintendo game called Gradius.  The game was a 2d side scroller, and you piloted a spaceship basically blowing things up.  For some whatever really weird reason there were enemies in the videos of moai shooting bubbles of their mouths. 

To this day, I wonder why these moais were even in Gradius. Perhaps maybe because of a story you will see later in this post?

Other than that video game, I really didn’t know much about the island at all.  Even guide books for some reason didn’t provide much information about Easter Island.

With the advent of the Internet, I think there definitely are tradeoffs to traveling.  On one hand doing research and learning about your destination is now easier than ever.  There is no shortage of travel influencers creating videos on Youtube about destinations you can watch.  With the Internet you can virtually experience almost everything before getting to your destination.

On the other hand, we have lost a sense of truly exploring the unknown and genuine wonder.  I remember backpacking through Europe in 2005 with really limited access to the Internet because smart phones were not widely available.  We got lost so often in the city, and nowadays with Google Maps, it is just really hard to get lost.

Getting to this tiny remote island was an adventure on its own.  You first had to get to Santiago, Chile, then you take a 5 hour flight directly west.  You are such in the middle of nowhere in the ocean that you can take a direct flight to Tahiti and then fly back to North America. 

After landing and settling in our hotel we had a day to explore the tiny town as our tour didn’t begin until the next day.

The first huge surprise was seeing everyone dressed up in the streets in costumes like Batman, The Flash, and even anime characters like Naruto.  After seeing they were getting candy from all of the store owners, we realized it was Halloween, Oct 31^st.  I asked how long this tradition has been around Easter Island and some shop owners said it came probably 10 years ago.  I’m amazed how some of the weirdest holidays in the Western world can migrate all over.

The second surprise I had was I expected Easter Island to be way more touristy.  In the town, barely anyone spoke English, so those old Spanish high school lessons were fortunate to help us survive going around town with Jason.

Kava Kava

To tour or not to tour?  Often this is a question always asked when you visit a foreign country, but for Easter Island, the choice is taken away from you.  Now when you visit Easter Island, you can only visit sites if you are going with a tour or through a local indigenous guide.  Lonely Planet recommended the tour group Kava Kava, so we booked a 2.5 day tour with them.

Our tour guide was Sebastian, and we were super lucky as he is one of the owners of Kava Kava.  In addition to being a tour guide, he formerly was a park ranger, and involved in the tourism ministry for the government.

The first fact we learned from him was that the name Easter Island came because the early colonizers discovered the island on Easter day.  However the local indigenous people call the island Rapa Nui (which is the same name as the indigenous population).

La Pandemia

We all were affected by the pandemic in different ways, but Rapa Nui really was a rough situation.  When borders were shut, the people on the island had to fend and survive mostly by themselves as the island was cut off from the world for about 1.5 years. 

In a normal year, the people of Rapa Nui would get their goods from the bellies of commercial flights, but once that dried up due to the closures, the people had to fend for themselves.

Many of the Rapa Nui people reverted back to their roots of fishing and farming, but many of the Chileans on the island didn’t have that ancestral knowledge to rely on and took free repatriation flights from the government to leave.  The population went from 10,000 people down to 5,000 people.

There was quite a bit of fear of reopening borders for quite a while until Omicron started. Once most people on the island got infected, and there weren’t any deaths, the island finally decided to open in 2022 with the requirement of a negative PCR test. However I think as of this writing, tests are no longer required to enter the island.

One thing which I think changed the experience of Rapa Nui for us was being on the island without many people.  Since at the time they just started to open up to the world, about half of the restaurants and the town generally speaking were understaffed.

Touring

Sebastian could write a whole book about the history of moais and the Rapa Nui people if he wanted to.  The most interesting thing to me about the trip was a reminder that history, contrary to our thinking of it, is not absolute and not set in stone.

Sebastian told us there are 3 important parts of the moai.  First, the ahu, the ceremonial platform the moai is on.  Secondly, the actual moai statue. And lastly the pu’kau, the hat on top of it.

We spent the first day visiting the Rano Raruku – the moai factory.  Moais were painstakingly handcrafted out of a mountain, and then moved up to 11 miles away.  In a totally different part of the island, the pu’kaus were made and then transported to the moais and placed on top of their heads.

Since the island just recently opened we had most of the site to ourselves. It was quite surreal to walk through moais through various state of restoration.

The current theory is that 4 people with 4 ropes were used to ‘walk’ the moai.  With one person on every corner they moved the moais slowly.

Pu’kaus on the other hand are still a big mystery.  To this day, archeologists, don’t have a settled theory on how these huge hats are placed on the moais. The further the mystery of moais, they are unable to carbon date the moais because there is no organic material on it.

The thing which moved me the most was that these people spent most of their lives making the moais, moving them, and repeating this process. Talk about a legacy because we are still admiring these statues 11 centuries later.

Sebastian said that most people ask how long it took to build the moais. He said that in our current capitalistic society that time is the unit of measure which most interested in. Back in the day the Rapa Nui cared of no such thing. I think there is something to be learned from this, where perhaps time shouldn’t be the most important factor, but instead the quality and the journey of what we go through.

I look at my life today and find myself so distracted with so many things I want to do.  Even in my Google Chrome tab, I probably have 50 tabs open with 3 instances of the browser to show the extent of my multi tasking of what is on my mind.

Looking at how the people worked on the moais have got me thinking, what kind of legacy do we want to leave?  And does making whatever legacy require a singular focus? Learning about the history of the Rapa Nui people revealed a beauty of doing one thing well, like really well, for a long time.

The Most Run Down Site

Towards the end of the tour, Sebastian took us to Ahu Te Peu (which oddly is on Google Maps and oddly is rated 4.1) where all the statues were knocked down and still on the ground.  It was definitely out of the beaten path where we had to hike there and nobody else was there at all.

He said that archeologists have specifically not restored the site to continue to do research on the moais.  I never really thought about it, but most of the historical things we see, look in awe, and post pictures on Instagram are of restored sites. Whether it be sites in Europe, pyramids, or temples, if things weren’t restored you would just see rubble.

As we walked around and looked and everything in a decrepit state, I found this moai site to be my favorite on the island.

Very rarely in life do we see something in its raw, unfiltered state.  Half destroyed, but preserved, and probably never restored again ever in the future.

Japan and Rapa Nui

We were standing by Ahu Tongariki, a set of moais by the ocean. Sebastian explained to us in 1960 a tsunami caused by an earthquake swept the moais way off the platform and all over the place. Since these statues were super huge, specialized cranes had to be used to move it because of its weight.

The Japanese company Tadano had the idea to volunteer to help restore the moai for their marketing material. The idea was that the company would benefit from the press of having one of their cranes restore such a historic site. The goal was to use the cranes and the moais in their marketing material.

As the Japanese worked on the restoration, they ended up learning of the spiritual significance of the site and withdrew all of their marketing materials out of a sign of respect. From that time forward, the Rapa Nui people and Japan have had a special relationship.

In current times, there has been much discussion of repatriation of stolen artifacts from colonial powers. In the British Museum, in the front is a moai where the Rapa Nui have petitioned the British government for its return.

In one sense, museums provide people the ability to experience artifacts and cultures without traveling afar. On the other hand, there is a troublesome history of artifacts stolen from other countries not being returned. In the perspective of the British Museum, once that floodgate opens, then all the countries will start asking for their goods back.

Sebastian told us the Japanese also are helping the Rapa Nui people recoat all the moais every 15ish years or so as the moais are indeed falling victim to weather and time. That eventually there will be a time they don’t exist anymore.

Since the Japanese and Rapa Nui have this special relationship it was interesting to learn that the Rapa Nui volunteered loaned one of their moais to Japan for an exhibit in Osaka in 1982. All this kind of reminds me of the importance of earning respect of cultures we encounter, and the special bonds and relationships that can form from mutual respect of one another.

San Pedro de Atacama – Explora

The Rapa Nui trip was so fulfilling in every sense that if we ended our Chile trip there, we could have just went home happy.  Fortunately we had another leg of the trip, the north of Chile, San Pedro de Atacama.­

Talking with Mark earlier, he mentioned that this area either serviced low end backpackers, or high end fully established excursion experiences.  Given there wasn’t a mid-range end, we opted to splurge on the latter. We chose Explora as they planned all the tour guiding for you.

After departing the plane and being exposed to the elements, it is as if a machine sucked all the moisture from my skin.  I knew San Pedro de Atacama was a desert, but I didn’t realize how harsh it was.  If you have ever been to Death Valley National Park, it is similar to that, but more intense.

Looking at the UV forecast, it was at 14 and uhh, I always thought 10 was the highest number.  After an hour via ground transport, we arrived at the hotel.

Explora is one of those experiences where every one there is on the fancier side of the income spectrum.  We met quite a few people there for their honey moon and chose Explora to not worry about the hassle of planning.

A trip planner helped plan our next 4 days.  We did a mix of hiking, and photogenic driving tours around the city.

The tours we went on were really great and were made in  way to maximize enjoyment.  For example, you would start at one part of the hike, and they would meet you at the end of the hike with a picnic.  In a regular hike you would start at some beginning point, but afterwards you would have to come back.

I had mixed feelings about the experience, as it was super nice, but only available to those with financial means to take these tours.

When I was back in Southern California recently visiting my parents, I met up with an old friend who was also in the computer science program at the University of California, Irvine.  After playing a tennis set, we were just chatting about our jobs and we talked about salary transparency.

I asked if we had an alumni event, would he share his salary with our friends from our graduating class.  He said yes, and we chatted a bit about how our generation is a bit reluctant to share salary with others.

I think within our social networks we probably can guess how much other people make through some visual cues (cars, lifestyles, housing, etc), but if we knew exactly how much our friends made would it affect our relationship with them?

My friend mentioned something super interesting where at his company, generation z employees are transparently sharing their salary with each other, regardless of the gap of salaries present amongst their peers in order to have the most information to see if any of them are getting underpaid.

I think there is a fear in letting people know how much you make.  Fear that perhaps it might change the relationship, but perhaps some transparency might actually help in people make a decision when pondering career paths.  I don’t know, this is really still an unresolved topic in my mind.

Rare Plants

The most memorable tour of the trip in San Pedro de Atacama, was the scariest one.  We started a hike pretty high up, about 4,000 meters to see the Tatio Geysers.  The geysers were super hot and cool to see, but the coolest part was starting a hike there along a hot river which was 40C.  As we walked along the river bed we saw this yellow grass which only exists above 4000m.

Towards the end of the hike we saw these green fuzzy things. Our hiking guide said they were called llaretas, and grew one mm a year. He guessed that this fuzzy plant was probably a thousand years old.

I just stood there in awe at something which could survive for such a long time. When I reflect upon this time frame, I wonder possibility what things we have in our society today will last a thousand years?

The things we collect will inevitably go out of date and decay, but I think nature will reign supreme in the long run in reminding us what is important with its longevity.

Valparaiso

The last part of our trip was visiting the city of Valparaiso.  I had wanted to visit the city (which is about an hour away from Santiago), last time I was there in 2015, but had gotten massively sick from some food at a restaurant back then.

Okay, this is my theory, but Jason disagrees.  Back in 2015 we were eating at Astrid y Gaston in Lima, Peru where we ordered the tasting menu.  There were four of us there, and we had one vegetarian menu as he didn’t eat meat.  Protein wise, we had raw fish, chicken, and some goat.

After the meal, the 3 people who ordered the meat tasting menu all got bad cases of diarrhea, while the vegetarian friend was fine.  Jason said it was because the vegetarian friend was from India and had a better microbiome, but I really just thought we got nailed from the meat.

This was an awful situation because I was going to hike the W, which was a 5 day hike in Patagonia Chile about 4 days later.  After a couple days of suffering with diarrhea, I gave up (as well as the other 3 meat eaters), and took antibiotics to clear out our system.  Seriously, after that experience I considered going vegetarian.

When we flew back from San Pedro de Atacama to Santiago, we took transit directly to Valparaiso.  The city is known for being completely covered with graffiti art.

Our tour guides in the city were Sebastian and Esteban.  We learned about the art of the buildings, and really the art on the buildings reflected either a love letter to the city or a statement of social protest.

This piece describes a protest against big farms taking water from all the smaller farms. Everywhere you turn left or right there is literally graffiti art somewhere. Even for local businesses, they request local artists to paint something local and relevant to their businesses on their storefront.

As with all informal economies, Sebastian told us the unwritten cultural rules of graffiti art. First rule is that artists don’t paint over each others art. If you did it and people found out, you would get a bad rap.

As we drove through byzantine streets, Sebastian stopped to explain this piece of art. What you don’t see is on the right of the building is the ocean front. This is an ode to locals who simply enjoy and never take for granted the sights and scenery of town. Many of the people on murals would be local people, like the old local milk man, newspaper delivery person, fisherman, etc.

This piece of graffiti art is an ode to refugees. As we finished the tour I didn’t actually think of anything as ‘graffiti’ as we have in our urban cities. Instead I saw everything as art, and perhaps this type of guerilla architecture and design is what we need in our hyper planned cities we have in North America.