Load your BigQuery Warehouse for custom analytics & modeling [Playbook]

BigQuery is a service by Google Cloud that provides fully managed warehousing with an enterprise relational database and supports client connections with numerous applications, such as reporting tools, analytical tools, enhanced business intelligence (BI) applications and ML-modeling tools like Vertex. BigQuery can operate on a petabyte scale if needed and allows you to query large amounts of data in multiple-stage operations. Data is backed up by very efficient storage and optimum query performance solutions working through a massively parallel processing and query execution.

In this use case guide, Lytics breaks down how you can connect Lytics to BigQuery to load BigQuery with all of your customer data and enable custom analytics and ML modeling.

Step 1: Align to your business needs

Lytics customers typically connect multiple data sources to Lytics to create unified profiles for segmentation and data activation. Most companies will use Lytics SDKs for web and mobile data collection in both iOS and Android apps, along with data from multiple third party applications and tools, including email, CRM, support along with back-end data from a data warehouse or transactional data from a digital commerce system like Shopify.  

A common challenge many businesses face is a multitude of data coming in from different sources in different formats and specifications. This requires data teams to spend a lot of time and resources building pipelines that convert and normalize the data in real-time into a suitable format for analytical or business needs. Lytics Conductor automatically adapts data collected from different sources to a consistent format, relieving those data engineering resources and increasing the speed with which teams can analyze and activate customer data.  

Customers have full control to create, manage and update a standardized, global customer data schema.  You can also create and manage an identity resolution strategy with Lytics, determining how you want to stitch together identifiers and join profile data.  All of these rules are customizable and can be updated by the user as business needs change.  

Step 2: Create and configure a BigQuery instance

Whether you already have your BigQuery instance or not, the first thing you need to do is to create a new principal  and a new service account in order to  set appropriate permissions. The reasoning here is simple. You would want to ensure that every incoming data agent with access to your data storage is strictly regulated, isolated, and monitored to decrease the chances of unintended data access. With a separate database user, the Lytics agent will only access the allowed resources within your data warehouse. Any data points will be out of reach and safe from potential threats.

Here is the Lytics step-by-step manual that will help you right away.

Step 3: Connect Lytics to BigQuery and enable BigQuery warehouse export connection

After successfully integrating the configuration in the previous step, you can connect and export specific tables to BigQuery, including but not limited to: 

1. Export Events – Export any or all of your event fields that come in from any data sources
2. Export All Users/Export Audience  – Export a full table of all your unified Lytics profiles or a specific audience of users , including  and desired IDs and attributes
3. Export Audience Changes – Export a table of all events that cause a user to enter or exit an audience segment state (i.e. entered “likely to purchase”). 

For these export options, you can follow the detailed instructions for the BigQuery connector.

Step 4: Test, configure, and verify 

Now it’s time to test the connection. Fortunately, Lytics has built-in functionality that helps with that process to view the specifics of how data will be treated in BigQuery. You can change the strategy for table data splitting, set up a custom threshold, switch the automated data hygiene that will auto-purge all expired data from the Data Warehouse, and even configure the delays between each data block load. 

The Lytics Jobs interface will help you understand potential problems via interactive error messages guiding you through invalid permissions levels, incorrect access credentials, and any other issues with the connection. 

When the test is done, you also need to choose the preferred export frequency.  The default export frequency is once per day.  

Step 5: Build custom data analytics dashboards & predictive models based on customer behavior & attributes 

By unifying customer data from Lytics with the data you already have in your BigQuery data warehouse, you can do lots of amazing things with SQL, Vertex, Looker and other BI/visualization tools. 

For example, you can:

• Analyze the best media channel for users based on your customer data and your media performance data
• Analyze number of qualified opportunities by channel – combining Salesforce opportunity data with website and Marketo data to identify which channels and campaigns drove the most leads.
• Visualize your customer’s paths from one journey stage to the next 
• Visualize a 360, comprehensive view of all of your customers with unified identities and unified data attributes
• Create custom predictive models to determine which of your prospect is most likely to purchase
• Create custom retention models to determine which of your customers is likely to churn

Use all of the data sources integrated through Lytics on your customer and leverage them to see the whole picture of the user within your BigQuery warehouse and your visualization tool.  

 Here’s a Looker dashboard example:

Step 6: Activate

Take the insights built out about your customers and pass them back into Lytics for improved campaign performance. By utilizing Lytics Cloud Connect you can quickly and dynamically create audiences using SQL in a user friendly interface within Lytics. Cloud Connect will constantly look for changes in audience membership and update corresponding profiles. Lytics can also import data using batch processing.

For these import options, you can follow the detailed instructions for the BigQuery connector.

You are just a few quick clicks away from a more efficient data driven campaign strategy.