Why we’ve sought a powerful data integration solution for customers
In today’s data-driven world, businesses and organizations are facing an ever-growing need to extract and process vast amounts of data from various systems. When it comes to handling large volume data—especially for ETL or ELT use cases—automation workflows gets complicated, creating issues for users such as:
File size limits
Slow processing speeds
Increase in the number of jobs and tasks consumed
The pain of designing and maintaining complex recipes
The administrative hassle of getting approvals for external storage systems like Amazon S3 and Google Drive
Additional cost of these storage systems
These problems are the reason why we at Workato have been working to develop FileStorage—a powerful, in-house storage and data integration solution that empowers our customers to store, manage, and process large files in recipes with ease.
How FileStorage works
A feature that allows you to handle files created or processed in your recipes, FileStorage provides the flexibility to store files as long as you need them, or delete them with the “Delete file” action via the FileStorage connector.
To start using FileStorage, use its connector in your recipes.
Note: access to Workato FileStorage is currently limited to select customers. Please reach out to the Workato Customer Support team to express your interest.
Search for the “Workato FileStorage” connector in the recipe editor
If a specific use case doesn’t come to mind, check out our community recipes to get ideas on how to start using it.
In this example, we’re getting leads from Marketo in batches, and storing them on FileStorage
Manage and preview files from the FileStorage UI
To view and manage the files you’ve created with the FileStorage connector in your recipe, go to our brand new FileStorage UI page, where you can easily:
See all the files and directories (folders) you created from recipes
Organize and search your directory
Quickly preview your CSV files for easy checking
Download files
Find this page under the side bar. Select “Tools” > “FileStorage”.
Note: only admin or analyst roles have automatic access to this page. For users with other roles, access must be granted by their admins.
Why you should start using FileStorage
FileStorage is a data integration solution designed to help our customers store their data on Workato, so fetching it and using that data across jobs and recipes become way easier, making it possible to:
1. Move large amounts of data quickly
For customers who need the flexibility of handling data at any volume, volatility, speed, source, destination and type, FileStorage helps them achieve this with an elastic architecture that automatically scales up or down to meet the workload. This removes the need to plan capacity, and provision or optimize for peak loads.
2. Transform large volumes of data with ease
It’s common for our users to transform tons of data on Workato, but designing recipes for this purpose can get complicated, making maintenance tedious too. Now, with SQL Transformations and Filestorage, transforming data at scale has become significantly simpler.
In the example below, we compare lead information from two sources: our current on-prem system, and cloud CRM leads housed on Google Drive, to capture changed data.
At the same time, we validate our leads’ personal email addresses. Finally, we bulk import our leads to Marketo.
What the recipe looks like without the use of FileStorage
With the addition of FileStorage and SQL Transformations (see recipe below), the same outcome can be achieved with just 3 actions. This also reduces the total time taken to validate and cleanse 6 million rows of lead information, from 6 hours to just 56 minutes!
Also watch: Advanced data transformations with SQL, Python, and Javascript
By leveraging the FileStorage connector, the same outcome is achieved with just 3 actions in much less time.
3. Simplify recipe design for lower task count
By using the FileStorage connector in your recipe, recipe design can be simplified to just a few actions. With less steps in a recipe, less tasks are consumed—allowing our customers to get more out of their Workato plan.
4. Maintain recipes with less effort
With simpler recipes and less reliance on external storage systems, less maintenance work like troubleshooting, testing and resolving errors is needed—freeing up the team to focus on the other things that need their attention.
Leveraging the above benefits might even save you the need to hire that additional headcount you’ve been thinking about.
A great data integration solution allows you to freely extract large volumes of data from a variety of systems and easily store it in a repository. It should also be completely encrypted, and allow customers to “bring their own keys” (BYOK) for easier security control.
FileStorage is this and more. It’s storage and data integration capabilities on an automation platform. So you can easily analyze, process and transform this data with what we call “recipes” at Workato. Ultimately, this allows organizations to transform data more efficiently and at scale.
Got questions? Check out our FileStorage docs page or directly approach your Customer Success Manager.
The post Build scalable data pipelines using Workato FileStorage appeared first on Workato Product Hub.
Read MoreWorkato Product Hub
