mohannadibrahim.dev

README.md/case studies/c4 ameritas retention

Case study 04 / Machine learning Pilot

Flagging flight-risk agencies about six months early

An agency-retention model built on public filings, with a Power BI Hit List that shows which agencies are likely to leave. In a 3-month regional pilot, 3 high-producing agencies were onboarded.

At a glancec4 / ameritas

Role
[CONFIRM: role on this project]
Client
Ameritas
Data
22,000 records, ~1,500 agencies, 4 states
Dates
[CONFIRM: project dates]
Status
3-month regional pilot
Result
$650,000 new annualized premium, attributed
Stack
Python, PyCaret, LightGBM, Power BI
Model
0.81 AUC
On this page
  1. Problem
  2. Constraints
  3. Approach
  4. Decision
  5. Evaluation
  6. Result
  7. What I'd do next

01Problem

Agencies that leave are expensive to lose, and by the time one is gone the chance to keep it has passed.

The goal was to see flight-risk agencies early, about six months ahead, so there was still time to act on it. [CONFIRM: how the business tracked agency attrition before this model]

02Constraints

  • The training data came from public sources only: DOI filings, NIPR, and M&A releases.
  • The people who would act on it needed a view they could open, not a notebook: a Power BI report.
  • [CONFIRM: any other constraints, such as data refresh cadence, compliance review, or who owned the pilot]

03Approach

Four steps, from public records to a list someone can act on.

  1. Collect. Public DOI filings, NIPR data, and M&A releases.
  2. Assemble. A 22,000-record training set covering about 1,500 agencies in 4 states.
  3. Model. LightGBM trained through PyCaret, reaching 0.81 AUC.
  4. Deliver. A Power BI "Hit List" that flags flight-risk agencies about six months early.
SOURCES Public records DOI filings NIPR, M&A releases TRAINING SET 22,000 records ~1,500 agencies 4 states MODEL LightGBM via PyCaret 0.81 AUC POWER BI Hit List flight-risk agencies ~6 months early 3-month regional pilot: 3 high-producing agencies onboarded SOURCES Public records DOI filings, NIPR, M&A releases TRAINING SET 22,000 records ~1,500 agencies, 4 states MODEL LightGBM via PyCaret 0.81 AUC POWER BI Hit List flags flight-risk agencies ~6 months early 3-month regional pilot: 3 high-producing agencies onboarded
Fig. 1From public records to a list someone can act on. The model scores agencies, and the Hit List is how the score reaches people.

04Decision

LightGBM through PyCaret, delivered as a Power BI Hit List. The model produces a score per agency, and the Hit List turns that score into a short list of agencies worth a call, about six months before they are likely to go.

Alternatives considered
[CONFIRM: other model families tried]not taken

[CONFIRM: what was compared and why it lost]

[CONFIRM: other ways to deliver the output]not taken

[CONFIRM: for example a spreadsheet or an alert, and why Power BI won]

LightGBM via PyCaret + Power BI Hit Listchosen

0.81 AUC on the agency data, delivered where the business already looks at reports.

05Evaluation

The model reached 0.81 AUC.

Placeholder[CONFIRM: how the 0.81 AUC was measured (split, time window, holdout), and the precision on the Hit List].

06Result

$650,000new annualized premium, attributed

In a 3-month regional pilot, 3 high-producing agencies were onboarded. The $650,000 of new annualized premium is attributed to that pilot.

07What I'd do next

  • Track the model after the pilot. Watch whether the Hit List keeps flagging the agencies that actually leave, and retrain as the filings change. [CONFIRM: any monitoring already in place]
  • Widen the footprint. The training set covers 4 states, so the next step is testing it beyond them. [CONFIRM: plans, if any]
  • Show how the premium was attributed. Document the link between the Hit List and the $650,000 so the number can be checked. [CONFIRM: attribution method]