Wednesday, June 19, 2024
No menu items!
HomeData IntegrationA Very Strange Year in the Data Business

A Very Strange Year in the Data Business

A very, very long year ago, I wrote about three scenarios for adjusting your data practice with respect to the impact of the pandemic on their businesses, and data practices. With 12 months plus now behind us, and cautious optimism on the horizon, it’s time to look back at what predictions did and did not pan out, and new learnings from a year like no other.

3 Scenarios for Doing Business in 2020

The 3 scenarios I laid out in April 2020 were:

The overnight digital transformation: organizations which were operating with roughly the same level of demand, but underwent a physical-to-digital transformation over the course of a few days, weeks or months.
The temporary hibernation: businesses which experienced a huge drop off in demand, and had to make significant cutbacks to try to survive.
The overnight spike: groups who saw a spike in demand and had to race to make the operational changes needed to fulfill that demand while still adhering to safety protocols.

So what have we actually seen happen? We saw some businesses take off and grow far more rapidly than they otherwise would have. Some recovered from an initial dip and returned to some new normal. Some are still struggling, and some had no choice but to shutter. 

The K-shaped Recovery

One thing that I did not expect was the so-called K-shaped recovery. The K-shaped recovery describes the vastly different economic trajectories of higher income knowledge workers who have been more able to work from home, are less likely to have lost their jobs, and saw their asset value grow in aggregate. In contrast, low-income workers, particularly in service jobs such as hospitality and retail, are much more likely to have lost their jobs, and faced far greater health and economic impacts from the pandemic.

Source: U.S. Chamber of Commerce, Sept. 2020

It was easy to predict that the “haves” in our society would fare better during COVID than the “have nots”, but the growth of the gap between the 2 into a chasm across many dimensions has indeed been shocking.

This K-shaped recovery extends beyond individuals to businesses. It was fairly obvious at the outset that certain industries (delivery services, telecommunications, digital entertainment, PPE providers) would accelerate due to COVID-19, while others would be hit hard (travel, brick-and-mortar retail, personal services). This K-shaped recovery has been a global phenomenon, seen in a variety of different economies.

Large vs Small Companies

But the K-shaped recovery split along other, less expected dimensions. Larger companies are more likely to have weathered the storm vs. small mom-and-pop businesses in the same industry. In part, they had more resources to act as a financial cushion and the know-how to navigate pandemic complexities such as applying for government loans.

White-owned vs Minority-owned Companies

In the U.S., even similarly sized small businesses are more likely to have received government aid if they were white-owned vs. minority-owned. Systemic barriers including access to the mainstream banking and legal systems, technological savvy and language blocked access.

Digital Transformation Maturity

As many predicted, the pandemic greatly accelerated digital transformation at organizations, large and small, across a wide range of industries. From financial services to online retailers to software, companies with a significant digital presence, fared best. 

Amongst the others, success often depended on how quickly they could become far more digital. Entertainment and event companies switched from in-person to virtual. Healthcare providers pivoted hard towards telemedicine whenever possible. Retail brands switched from brick-and-mortar distribution to social influencers and online channels. Then there were the big box stores that finally were able to take on Amazon by combining digital and brick-and-mortar into reliable curbside pickups.

7 Lessons Learned from the Data Integration Front Lines

Amongst our customers here at StreamSets, we said goodbye to a handful of customers that shut their doors. We look forward to crossing paths again with the talented data professionals we worked with at these companies. 

We also saw surprising resiliency in companies in hard-hit industries such as travel and live entertainment. Even though their business cratered in the early months of the pandemic, they were able to cut costs, pivot towards digital, and achieve a slow recovery. This would not have been possible without the flexibility and data insights to go into an economic hibernation mode, retool operations, and then re-emerge perhaps smaller but more resilient. The experience was not without pain, but they survived and are ready to grow in the new world.

We have also seen companies in industries that have either held steady or grown during COVID. In another unexpected outcome, companies in highly regulated industries such as financial services, communications, healthcare and pharmaceuticals, greatly accelerated their digital transformation and business data initiatives, even if they didn’t face a pandemic boom or bust. 

1.  Virtualize as many operations as possible.

This trend became imperative in the effort to protect employees, customers, vendors and partners from the risk of COVID-19. But many companies will not return to pre-pandemic operations because these changes make sense in a highly digital world that demands agility and flexibility.

2.  Scale rapidly and non-linearly to manage demand.

Digital infrastructure is inherently designed to scale more effectively than infrastructure tied to manual operations and management. This applies to both scaling up and scaling down. It’s why so many companies are adopting DataOps practices.

3.  Match increased risk tolerance with agility.

Many companies discovered that they could go faster, and accept more risk when forced to change quickly. For example, the pivot to working from home for a large number of employees shook up many hide-bound companies. Staff working remotely didn’t bring productivity to a standstill, even while juggling home-schooling and other unexpected responsibilities. Many companies have taken far bigger steps in their digital transformation journey than they would have otherwise.

4.  Dial up real-time data to manage dynamic situations. 

With conditions changing week-by-week and day-by-day, leaders across all types of organizations have turned to the latest data to make the best decision possible. There was simply no time for complicated change management processes when new types of data from new systems had to be integrated into the larger data and analytics environment literally overnight. (State of Ohio did exactly that in order to create its COVID-19 dashboard.)

5.  Migrate as much as you can to the cloud. 

These rapid pivots kicked cloud migrations into a whole new gear. Large, traditional enterprises had been tip-toeing into cloud due to concerns about security and governance. In 2020, they plunged in head-first. Organizations that swore that they would never trust the cloud made huge bets on it. (Hear how BT, the UK telecommunications giant, migrated to the cloud.) For those who had to scale essentially overnight, cloud was the only option– a traditional data center build-out takes quarters or years. 

6.  Invest in new cloud data platforms for data lakes, data warehouses, and ML platforms. 

With skyrocketing demand for data and insights in the midst of constant change, businesses have jumped onto the latest and greatest data platforms. From cloud data lake integration to cloud data warehouses, just witness the growth in Snowflake, Databricks, S3, Redshift, Azure Synapse, ADLS, Google BigQuery and Dataproc. Again, what was already a big trend stepped up an order of magnitude due to the forcing function of the pandemic.

7.  Be kind, be patient. 

We couldn’t take our clients out to dinner or visit them in person. Instead we worked out challenges and contracts eye-to-eye through a Zoom call with conspicuous personal space in the background. Perhaps, we will all see each other with a bit more humanity and a bit more grace.

I’ve been in the data integration space for many years and experienced big transitions, but I’ve never been through anything quite like this past year in the data business…I’m optimistic for what is ahead. We’ve finally been able to unlock some of the big blockers to change and there is no turning back. 

The post A Very Strange Year in the Data Business appeared first on StreamSets.

Read MoreStreamSets



Please enter your comment!
Please enter your name here

Most Popular

Recent Comments