During Prelim QA, the Ops Team is checking that matching is complete and that all the data they collected or downloaded made it through the entire matching process and showed up correctly in the dashboard.

In general, Ops Team is reviewing data consistency on different Looker Boards.

Unless the investigation below discovers issues to investigate, this can be done in about 10 minutes per rooftop. The approach here goes from the most general view to the most detailed view for efficiency.

Step 1:  Check that All Leads were Loaded

1. Go to the Leads QA 2 board in Looker
2. Select client and property name
3. Set dates to last 3 complete months
4. Look at the counts on the Monthly Leads Report (with Property) and compare the latest month to prior months. 

What are you looking for?

• Confirm that all channels are populated
• Confirm that numbers are consistent with prior month
• Are any Lead Providers missing or added?
• Did the counts increase or decrease dramatically? Note that some degree of variation is normal, especially since it will usually track with sales increases or decreases.

If there are issues:

• Lead Provider file missing or added:  If any Lead Provider was dropped or added from the customer’s data for the prior month, this would have been known by the Ops Team during data collection and documented on the SA Processing board. If Looker reflects additions or subtractions that are not consistent with Monday.com, Ops Team will check to ensure that the Lead report file is in our folders. Since files move sequentially during processing from “Insyncing Data Drop” to “Data Archive,” Ops Team will check both:
  • If the file is not found, Ops Team will re-collect the file and load it
  • If the file is found, Ops Team will make a ticket for Jordan to diagnose why the file wasn’t processed fully.
• Lead Counts increased dramatically:  This may indicate that we picked up data we shouldn’t have, or even for the wrong rooftop. For example, we may have loaded both inbound and outbound calls, when we should only have inbound calls. Ops Team will look at the raw file to verify that we have the right data.
• Lead Counts decreased dramatically:  This may indicate that the file for the previous month was incomplete, or even for the wrong rooftop. It may be missing dates, rooftops, or lead types. Ops Team will look at the raw file to see if it looks complete:
  • If the file is not complete, Ops Team will check with the data provider to see whether we received a bad file or if there’s a good reason. For example, if the vendor was dropped mid-month, there wouldn’t be a full month of data. In the screenshot above, CarGurus dropped to only one lead, reflecting cancelation.
  • If the file is complete and lead counts are similar to prior months, Ops Team will make a ticket for Jordan to investigate.
  • If the file is complete and lead counts are significantly lower overall, Ops Team will make a note on the SA Processing board and tag the CSL for review. The customer may have changed strategy or services with the provider, affecting lead count.

Step 2:  Check Matching 

1. Go to the Attribution Overview looker board
2. Select client and property name
3. Set dates to last 3 complete months
4. If looking at multiple properties, sort Monthly Property Report by Sales Month first, then Property Name
5. Look at matching levels for the latest month and compare to prior months
6. Offline Matching less than 35% is a flag.
   1. Note in the conversation updates the reason for stopping. Note any other anomalies (such as low lead counts, low or high sales counts, etc.)
   2. If matching has dropped significantly (blocker):
      1. Stop and mark the client's Processing Status "QA Issue" on SA+TVA Ops Overview board
      2. Follow steps for investigation as outlined in Offline Attribution QA Issues.

What are you looking for?

• Review Monthly Property Report chart for consistency and adequate reporting
  • Sales Counts
  • Sales with Phone %
  • Sales with Email %
• Are any matching types missing?
• Are any matching types greatly lower than last month?
• Did the Attribution counts decrease dramatically? Note that some degree of variation is normal, especially since it will usually track with sales increases or decreases.

If there are issues:

• Overall drop in matching: Did sales drop? Attributed sales may be lower just because total sales are lower. The black line in the chart indicates the sales count, while the white line indicates the matched sales. The number chart below the graph, Monthly Property Report (Stage Environment), also indicates the % of sales with Valid Phone & Email, % of Sales Attributed, in addition to Non-Web Traffic, Web Traffic, Liveramp and Mediawallah %. Are these percentages proportional to last month? 
• Liveramp matches not showing up (or unusually low): This likely means that there hasn’t been enough time from when Sales were uploaded for Liveramp matches to come back. This timing can vary, with the usual duration of about 5 to 7 days, but up to as much as 22 days. 
  Ops Team does not QA LiveRamp or other Online Matching. It is part of CSL QA.
  • Other causes can include:
    • Property not setup properly in DMUI. Must have LiveRamp=TRUE and SA Visible=TRUE
    • Mapping issues with sales file - verify there is good PII and vehicle info in Sales Standardized
    • Failure of Sales Preprocessing, Sales Processing, ChartIO_MatView_Refresh (Looker refresh) or LiveRamp DAGs (check with Engineering)
• Declines in Non-Web Traffic Leads: This comprises CRM and Call Tracking data, and a large drop may indicate issues with Leads (wrong file saved, file not loaded). This can be investigated in the next QA step.
• Lead Attribution decreases to zero (or almost zero):  First, this should be consistent with the Lead Count in Step 2.  If the count is low and has been verified, then the attribution will also be low, but if the lead count is consistent and lead attribution is low, it could indicate that the wrong rooftop file was collected or processed, so accordingly none of it is attributed. In the screenshot above, the decrease for CarGurus is consistent with the drop we saw in Leads in Step 2.
• Lead Attribution decreases dramatically but not to zero:  The Ops Team will look into this to see if it’s reasonable given context:
  • Compare to the Lead Count:  In the screenshot above, Care Connect decreased to half the attributed leads of the prior month, but when we checked lead count in Step 2, the count didn’t decrease as much. So that didn’t explain it.
  • Compare to the Sales Trend:  Generally speaking, Lead Attribution will track with overall Sales Attribution. In Step 1 we saw a large decrease in sales (and attributed sales), so it’s reasonable that the attributed sales for Care Connect would drop.
  • If the attribution drop cannot be explained by sales trend or lead count, Ops Team will tag the CSL in the Conversation for the appropriate rooftop on the SA Processing board.

Step 3: Potential QA Issues

If Ops Team finds issues that need to be investigated:

1. Set Processing Status to QA Issue.
2. Investigate the root cause. See Researching QA Issues.
3. Reach out to the CSL by tagging them in an update on the SA + TVA Ops Overview board line item - under the task Conversation.

Step 4: Prelim QA Complete

If overall Ops Team has verified that all the files collected were processed and attributed normally, set Processing Status to Automatch In Progress. If the LR Done date has already hit, after setting to Automatch In Progress, set to CSL QA.