During Prelim QA, the Ops Team is checking that matching is complete and that all the data they collected or downloaded made it through the entire matching process and showed up correctly in the dashboard.

In general, Ops Team is reviewing data consistency on different Looker Boards.

Unless the investigation below discovers issues to investigate, this can be done in about 10 minutes per rooftop. The approach here goes from the most general view to the most detailed view for efficiency.

Step 1:  Check that All Leads were Loaded

1. Go to the Leads QA 2 board in Looker
2. Select client and property name
3. Set dates to last 3 complete months
4. Look at the counts on the Monthly Leads Report (with Property) and compare the latest month to prior months. 

What are you looking for?

• Confirm that all channels are populated
• Confirm that numbers are consistent with prior month
• Are any Lead Providers missing or added?
• Did the counts increase or decrease dramatically? Note that some degree of variation is normal, especially since it will usually track with sales increases or decreases.

If there are issues:

• Lead Provider file missing or added:  If any Lead Provider was dropped or added from the customer’s data for the prior month, this would have been known by the Ops Team during data collection and documented on the SA Processing board. If Looker reflects additions or subtractions that are not consistent with Monday.com, Ops Team will check to ensure that the Lead report file is in our folders. Since files move sequentially during processing from “Insyncing Data Drop” to “Data Archive,” Ops Team will check both:
  • If the file is not found, Ops Team will re-collect the file and load it
  • If the file is found, Ops Team will make a ticket for Jordan to diagnose why the file wasn’t processed fully.
• Lead Counts increased dramatically:  This may indicate that we picked up data we shouldn’t have, or even for the wrong rooftop. For example, we may have loaded both inbound and outbound calls, when we should only have inbound calls. Ops Team will look at the raw file to verify that we have the right data.
• Lead Counts decreased dramatically:  This may indicate that the file for the previous month was incomplete, or even for the wrong rooftop. It may be missing dates, rooftops, or lead types. Ops Team will look at the raw file to see if it looks complete:
  • If the file is not complete, Ops Team will check with the data provider to see whether we received a bad file or if there’s a good reason. For example, if the vendor was dropped mid-month, there wouldn’t be a full month of data. In the screenshot above, CarGurus dropped to only one lead, reflecting cancelation.
  • If the file is complete and lead counts are similar to prior months, Ops Team will make a ticket for Jordan to investigate.
  • If the file is complete and lead counts are significantly lower overall, Ops Team will make a note on the SA Processing board and tag the CSL for review. The customer may have changed strategy or services with the provider, affecting lead count.

Step 2:  Check Matching 

1. Go to the Attribution Overview looker board
2. Select client and property name
3. Set dates to last 3 complete months
4. If looking at multiple properties, sort Monthly Property Report by Sales Month first, then Property Name
5. Look at matching levels for the latest month and compare to prior months
6. Offline Matching less than 35% is a flag.
   1. Note in the conversation updates the reason for stopping. Note any other anomalies (such as low lead counts, low or high sales counts, etc.)
   2. If matching has dropped significantly (blocker):
      1. Stop and mark the client's Processing Status "QA Issue" on SA+TVA Ops Overview board
      2. Follow steps for investigation as outlined in Offline Attribution QA Issues.

What are you looking for?

• Review Monthly Property Report chart for consistency and adequate reporting
  • Sales Counts
  • Sales with Phone %
  • Sales with Email %
• Are any matching types missing?
• Are any matching types greatly lower than last month?
• Did the Attribution counts decrease dramatically? Note that some degree of variation is normal, especially since it will usually track with sales increases or decreases.

If there are issues:

• Overall drop in matching: Did sales drop? Attributed sales may be lower just because total sales are lower. The black line in the chart indicates the sales count, while the white line indicates the matched sales. The number chart below the graph, Monthly Property Report (Stage Environment), also indicates the % of sales with Valid Phone & Email, % of Sales Attributed, in addition to Non-Web Traffic, Web Traffic, Liveramp and Mediawallah %. Are these percentages proportional to last month? 
• Liveramp matches not showing up (or unusually low): This likely means that there hasn’t been enough time from when Sales were uploaded for Liveramp matches to come back. This timing can vary, with the usual duration of about 5 to 7 days, but up to as much as 22 days. 
  Ops Team does not QA LiveRamp or other Online Matching. It is part of CSL QA.
  • Other causes can include:
    • Property not setup properly in DMUI. Must have LiveRamp=TRUE and SA Visible=TRUE
    • Mapping issues with sales file - verify there is good PII and vehicle info in Sales Standardized
    • Failure of Sales Preprocessing, Sales Processing, ChartIO_MatView_Refresh (Looker refresh) or LiveRamp DAGs (check with Engineering)
• Declines in Non-Web Traffic Leads: This comprises CRM and Call Tracking data, and a large drop may indicate issues with Leads (wrong file saved, file not loaded). This can be investigated in the next QA step.
• Lead Attribution decreases to zero (or almost zero):  First, this should be consistent with the Lead Count in Step 2.  If the count is low and has been verified, then the attribution will also be low, but if the lead count is consistent and lead attribution is low, it could indicate that the wrong rooftop file was collected or processed, so accordingly none of it is attributed. In the screenshot above, the decrease for CarGurus is consistent with the drop we saw in Leads in Step 2.
• Lead Attribution decreases dramatically but not to zero:  The Ops Team will look into this to see if it’s reasonable given context:
  • Compare to the Lead Count:  In the screenshot above, Care Connect decreased to half the attributed leads of the prior month, but when we checked lead count in Step 2, the count didn’t decrease as much. So that didn’t explain it.
  • Compare to the Sales Trend:  Generally speaking, Lead Attribution will track with overall Sales Attribution. In Step 1 we saw a large decrease in sales (and attributed sales), so it’s reasonable that the attributed sales for Care Connect would drop.
  • If the attribution drop cannot be explained by sales trend or lead count, Ops Team will tag the CSL in the Conversation for the appropriate rooftop on the SA Processing board.

Step 3: Potential QA Issues

If Ops Team finds issues that need to be investigated:

1. Set Processing Status to QA Issue.
2. Investigate the root cause. See Researching QA Issues.
3. Reach out to the CSL by tagging them in an update on the SA + TVA Ops Overview board line item - under the task Conversation.

Step 4: Prelim QA Complete

If overall Ops Team has verified that all the files collected were processed and attributed normally, set Processing Status to Automatch In Progress. If the LR Done date has already hit, after setting to Automatch In Progress, set to CSL QA.

Once the Processing Status on the SA Processing board hits “CSL QA” or “QA Issue” it’s the CSL’s turn to check that labels are correct and that any customer requests from the last month (such as new classifications or labels) were addressed.

Step 1: Source Classification

1. Run the unclassified report and go through it and make sure all your properties are classified correctly. Note: you must have this file uploaded by 3 pm est daily. https://www.dropbox.com/home/Insyncing%20Data%20Drop/Source%20Classification/Classify%20These%20Files

Step 2: Call Tracking Classification

1. Run the unclassified report and go through it and make sure all your properties are classified correctly. Note: you must have this file uploaded by 3 pm est daily. https://www.dropbox.com/home/Insyncing%20Data%20Drop/Call%20Tracking%20Classification/Classify%20These%20Files

Step 3:  Check that All Leads were Loaded

1. Go to the Leads QA 2 board in Looker
2. Select client and property name
3. Set dates to last 3 complete months
4. Look at the counts on the Monthly Leads Report (with Property) and compare the latest month to prior months. 

What are you looking for?

• Confirm that all channels are populated
• Confirm that numbers are consistent with prior month
• Are any Lead Providers missing or added?
• Did the counts increase or decrease dramatically? Note that some degree of variation is normal, especially since it will usually track with sales increases or decreases.

If there are issues:

• Tag Ops and/or CSL in the item's conversation updates

Step 4: Attribution Overview board in Looker

1. Go to the Attribution Overview looker board
2. Select client and property name
3. Set dates to last 3 complete months
4. If looking at multiple properties, sort Monthly Property Report by Sales Month first, then Property Name
5. Look at matching levels for the latest month and compare to prior months:
   1. Review Attribution % (don’t look at Sum of Total Attributed Sales - raw #) 
   2. Review Offline Attribution %  [CRM, Chat, CT, Email Opens, Direct Mail]
      1. If problem, we may have loaded files wrong, mapped incorrectly, etc. - Look at Leads QA2 for the property - make sure the # of leads are in line
   3. Offline Matching less than 35% is a flag.
      1. Note in the conversation updates the reason for stopping. Note any other anomalies (such as low lead counts, low or high sales counts, etc.)
      2. Follow steps for investigation as outlined in Process for Investigating Low Offline Attribution
   4. Review Online Attribution %   [LFI (Clarivoy Leads), Lat/Long, Lat/Long 3rd Party]
      1. If problem, open a ticket for Engineering
   5. Review LiveRamp Attribution %

What are you looking for?

• Review the Monthly Property Report module and check across all columns for any major deltas (which may lead to issues)
• Look over data volume by Lead supplier/type for major changes (which may indicate issues with ingestion)

If there are issues:

• Tag Ops and/or CSL in the item's conversation updates

Step 5: Review Client data in Stage Dashboard

1. Attribution Overview
   1.  Look at how channels and sources are trending
2. Executive Summary
   1. Check the Executive Summary and make sure the data is correct.
3. Sales Overview
   1. Check Sales Overview make sure you have no unclassified sales.
4. KPI over Time
   1. Go to the dashboard, “KPI Over Time” page
   2. Select and deselect channels to look at how they change over time
   3. Click to sources and see how they are trending
5. Campaigns
   1. Go to the dashboard, “Campaigns” page
   2. Look through the “Identified Campaign” list and note how each campaign has been classified into Channel and Source
   3. What are you looking for?
      1. Do the campaigns look like they’re associated with the correct sources and channels? Sometimes this level of data uncovers classification issues that were not evident at higher levels. For example, a campaign label that says “CarGurus SEM” may be classified as Organic Search. 
   4. If there are issues:
      1. The approach to fix classification issues depends on what type of data it is. First you need to determine the type of data. Call Tracking data can be easily recognized because the campaign label will always include the phone number. If it’s not Call Tracking, then make a Manuscript ticket for the ETL team to find out if it’s CRM data or web traffic.
      2. Once you know what the data type is, you can look at the raw data to see if the increase is valid. CRM attribution and call tracking attribution can be checked in the respective raw files. Web traffic attribution can be checked in Google Analytics (if we have access) and the Web Traffic report to see tagging.
      3. If you find that a source is mis-labeled, follow the steps in the Taxonomy chapter to address.
6. ROI
   1. Look over the ROI page 
   2. Check for data splits for source consolidation errors - possible issues: 
   3. two sources for the same thing in one channel >> ticket for source consolidation 
   4. two sources for the same thing in different channels>>  this issue must be discovered: check source and call tracking - if no issues are reconcilable there, then check the taxonomy data and if the issue is web taxonomy >> create taxonomy ticket on the Support > Tickets Monday board 
      
       
   5. What are you looking for?
      1. Are the channels trending with the overall sales trend?
      2. Did any channels increase or decrease significantly from last month?
      3. Within channels, did any sources increase or decrease significantly from last month?
      4. If so, do you already know why? (e.g., client canceled direct mail)
   6. If there are issues:
      1. Click to the Source level:  If you see large changes, especially if they don’t track with other channels, click to the Source view to see which source is driving the change. Does that source belong there? Could that source be mislabeled? Sometimes this indicates that a customer has re-assigned a call tracking number but did not update the label. For example, a number labeled as “Facebook” might suddenly be re-used on the customer’s main home page, so calls will spike and are unrelated to Facebook. If you don’t recognize why the source has increased or decreased:
         1. Make a ticket for the ETL team to find out where the attribution increase is coming from. Is it CRM, call tracking, web traffic?
         2. Once you know what the data type is, you can look at the raw data to see if the increase is valid. CRM attribution and call tracking attribution can be checked in the respective raw files. Web traffic attribution can be checked in Google Analytics (if we have access) and the Web Traffic report to see tagging.
         3. If you find that a source is mis-labeled, follow the steps in the Taxonomy & Data Classification Process Document to address.
      2. Make sure you look at the channel marked “Other” to see what sources are classified there. This is often where you will find attribution for sources where the name was vague (for example, “Google,” which could be organic, paid or even display), which can then be clarified with the customer when they see the data. But sometimes you will find sources that are clear enough that they should be re-assigned to other channels.
      3. Check attribution for other rooftops in the group:  Since for dealership groups we run matching holistically across the group, you may find attribution showing up in one rooftop for a vendor that this rooftop doesn’t use. For example:
         1. The customer may be working with Autotrader for only one store, and an Autotrader lead submitted to that store ended up buying from another store. So there would be attribution to Autotrader at the second store even though they’re not using Autotrader.
         2. The customer recently doubled their paid search spend for one store (or even the parent domain) and this resulted in an increase in paid search attribution for other stores as well.
         3. You can see the nature and degree of this cross-family attribution by looking at the “Sales Overview” page and using the “Touchpoints” part of the Filter tab on the left side. By toggling the store’s domain off, you can see the sales that were influenced by other properties. Note: third party auto vendors will always be listed separately in this Touchpoint list, regardless of whether they provided a lead to this store or elsewhere.
            
            
            In the screenshot above, the domain for this dashboard (BaxterSubaru.com) has been toggled off in the filter, so all the attribution showing here is for Subaru sales influenced by other Baxter stores. Direct Mail, for example, shows up here even though the Subaru store does not use Direct Mail.

Step 6: CSL QA Complete

Once you complete QA, determine whether the data is ready for the client to see. 

• If everything is fine but matching is not yet complete, we usually want to hold it because the data may look low and be unsatisfying to the customer. Change the status to Waiting on Classifications if it needs more time for LR/NR” and hold it there. Make sure to check it every day so that when matching is completed.
• If you find errors in QA that are just misclassifications, we usually want to share it with the client anyway, while we work in parallel on the relabeling. If it’s large enough that the client will notice, you may want to tell the client proactively as a known issue that you’re fixing. Most of the time the issues are small, worth fixing but not worth mentioning to the client. By all means, though, use your judgment. If you find a source has been mislabeled and it’s creating a dramatic spike and it’s quickly fixed in either CRM labeling or Call Tracking, then you may want to fix it before pushing the data to Prod. Basically, ask yourself “if the client sees the dashboard as is, will it give them a reason to doubt Clarivoy’s competence or accuracy?” If the answer is yes, then hold the data until it’s fixed. Change the status to “QA Issue,, See Comments for QA issues and any updates.

The process of pushing data from Stage to Prod can take as long as 48 hours. Once it’s finished, the CS Specialist will change the status in Monday to Approved for Prod. The Ops Team will then push the client data into Prod and change the status to Prod In Progress. After verifying Prod has successfully run, Ops Team will change the status to Verify Prod.