A leading Open SaaS ecommerce platform’s journey toward Kafka modernization and real-time data analytics

When a company utilizes an outdated IT infrastructure, management and maintenance become more problematic as it scales. This often prevents businesses from acting quickly to embrace new opportunities for innovation. Fortunately, avoiding these challenges is now easier than ever thanks to the availability of proven cloud services.

A few years ago BigCommerce (NASDAQ: BIGC), a leading Open SaaS ecommerce platform for fast-growing and established B2C and B2B brands, faced challenges as it looked to expand and meet changing customer demands. The company was managing its own Kafka cluster that required increasing maintenance on software patches, blind spots in data-related infrastructure, and system updates.

BigCommerce chose to migrate to a fully managed Apache Kafka® platform to modernize its approach with the help of Google Cloud and Confluent. By doing so, BigCommerce increased system performance, reduced maintenance, and was able to spend more time developing next-generation services for their customers. 

A better approach to Kafka-based data streaming

BigCommerce endures a relatively unique set of technical demands given the broad scope of its ecommerce platform.

“We are an Open Software-as-a-Service (SaaS) ecommerce platform that equips mid-market and enterprise brands with a composable foundation to accelerate ambitious growth,” says Mahendra Kumar, vice president of data and software engineering at BigCommerce. “This means we have to deal with a lot of complex data across a range of commerce categories.”

BigCommerce recognized that Kafka was the right technology for its real-time data processing. As they continued to scale their Kafka footprint across the organization, the engineering team had to manage an exponential increase in data demands from a larger base of users.

“Three engineers spent part of their time on Kafka management making updates, patching and troubleshooting our Kafka clusters,” says Mahendra.

BigCommerce chose to migrate its data-streaming practices to Confluent Cloud because it provides everything needed to connect apps, data systems, and organizational frameworks for real-time data streaming and processing through a Kafka-based structure that minimizes maintenance and management demands.

 “Confluent was ideal for our needs because it frees us from the operational aspect of Kafka through the managed service offering. The fact that Confluent is a company founded by the original creators of Kafka gave us a lot of confidence in their ability to understand our use case and help us implement best practices,” says Mahendra. “We are very comfortable with Kafka principles, so we didn’t experience migration pains or deal with a steep learning curve. Confluent is reliable, robust, performant, and developer friendly. It also saves us a lot of money, while improving our Kafka practices. The Confluent support team is very helpful and is quick to respond to open support cases.”

At the time of migration, BigCommerce processed 1.5B events daily. These events are processed with greater efficiency thanks to Confluent. These events include product orders, website visits, page views, cart information, and others. The improved system performance greatly enhances merchant experiences with the platform, while also dropping resource demands from Kafka cluster management to the equivalent of about one half of a full-time engineer’s workload.

Unlocking data insights for merchants at scale

In addition to saving time and money, the combination of Confluent and Google Cloud proved to be the right solution for another major priority — providing merchants with the freshest, most valuable data analytics in real-time.

“Merchants need actionable real-time analytics to remain competitive in their respective marketplaces. We set out to build a data platform to meet those requirements,” says Mahendra. “Open data is key to our Open SaaS strategy. BigCommerce provides native integrations for merchants to automatically upload their commerce data into BigQuery daily to build custom reports, integrate their commerce data with other sources such as ads and CRM, and unlock the power of that information.” 

The highly-scalable, fully-managed features of Google Cloud and Confluent enable BigCommerce to more reliably support merchants regardless of spikes in demand, such as major shopping events like Cyber Monday.

“With Google Kubernetes Engine (GKE), Cloud SQL, Cloud Composer, and Confluent, we have more confidence in our Kafka streams and real-time data platform at any time during the year. We no longer have to worry about scaling because it’s all automated,” says Mahendra.

By running its storefront platform on Google Compute Engine and deploying a new data analytics platform using other Google Cloud technologies, BigCommerce has saved both time and costly investments on data transfers. Today, BigCommerce allocates fewer resources on backend infrastructure management allowing it to focus on increasing the power of its platform for merchants.

Learn more about how Google Cloud partners can help your business achieve a successful digital transformation. Get started with Confluent Cloud for free today, and get $400 in free credits to spend during your first 30 days.