A good metric is like a ship’s compass that can be used to mark a crew’s position, warning them if they’re veering off their bearing and validating that their efforts are leading them to where they want to be. A metric, like a compass, is an aid that when used with other information, can verify your ship’s position. For those of you who are pirates, this means you knopw where you are on that treasure map, and you're closer to that precious booty.

But like a compass that hasn't been calibrated to the ship’s magnetic idiosyncrasies, or the magnetic deviation of the area, a metric, can give a team false perceptions. It’s important to ask the following two questions.

1. What other signs indicate my report is accurate?
2. How does the metric relate to your customer's feelings?

Let me share two examples of when I was completely off the mark with both these points.

During my time at Europe’s biggest mid-term rental marketplace HousingAnywhere, I had the pleasure to work on a feature that we called HousingAnywhere payments. HousingAnywhere payments allowed property managers to automate their rental expanse payment collection. Essentially property managers could request money from tenants through our platform. This would typically be things like damage deposits and monthly rent.

Whenever we talked about HousingAnywhere Payments feature we’d refer to a metric that we called the success rate. We would use a guardrail metric we called the payment success rate to make decisions about how successful the product was in general.

Here’s how it was calculated;

Everything seemed fine right, nice and simple calculation. But keep reading to learn about how this became an issue.

Throughout some analysis, we found that property managers were increasingly creating payments due long into the future. So in response, we introduced a new feature that allowed property managers to set up an entire payment plan in one action. This action was typically undertaken by a property manager at the beginning of a tenancy. They would set everything up and let HousingAnywhere be responsible for collecting their money from the tenant.

Not long after launching this feature, we saw a meteoric drop in the payment request success rate. Although, this drop wasn’t reflected in any of our other reporting. Our payment volumes and usability metrics remained within their control limits. It was soon clear that there were no other signs to verify this report was working as we expected.

So how did we explain the drop in success rate? Too much rum clouding our judgement? Or Scurvy maybe?

Well, as you remember from the calculation above, payment collection was compared on a created vs paid basis. This worked fine when payments were due immediately, but broke when property managers created payments to be paid in the future, some up to 12 months into the future. Imagine a property manager creating 12 payments, one for each month’s rent for an entire year, but only one would be due immediately. You may have already realised that this would increase the denominator, and regardless of the numerator, leads to a much lower value. That explained our success rate drop.

This then forced us to ask ourselves what success for a property manager truly was before proceeding. In other words, how does what you’re reporting relate to the customer's feelings?

So after researching some of our key property managers, we found that the most important customer feeling to attribute success to was having their rental expenses paid on time. In other words, not collecting the money on time should be considered a “failure”. This definition became the basis of our new metric.

This meant that instead of basing everything on payment creation, it would need to be based on the payment due date.

This is when we conceived the “payment conversion window” approach. The concept was simple, all payment requests would have a payment conversion window. When it opens - and we expect the payment to be made soon, and when it closes - and when the payment should have been completed already. Then all that’s required is looking at all the payments with their conversion window closed for a timeframe, and checking if they’ve been collected or not.

We had a metric! but now we wanted to show this metric to stakeholders visually. We needed a report.

We decided a stepped conversion funnel would lend itself best to this metric. But this came with two hurdles to overcome, mostly on the Mixpanel side of things.

1. The conversion window couldn’t be based on the window closing, it had to be based on the window opening. In Mixpanel, to make funnels, product people define two events that happen in succession (or not). The first event starts the tracking, the second measures the success. In our case, the funnel was `all the payment windows that opened that were followed by a payment` instead of `those payments collected that were preceded by a payment window opening`. But luckily the approach of starting with the window opening came with a bonus. By not declaring a payment conversion window close time, we had the flexibility to change this in the future via Mixpanel directly without needing engineers.
2. It was also important to make sure the granularity was set to the individual payments, and not the users. The report needs to show the number of absolute payments and not the users with payments. Keep in mind that Mixpanel sets users as their main object. Whenever anything happens your application sends an event to Mixpanel with the user’s id (if it’s available) plus the data on the interaction. We had to send the payment’s unique id so if a payment was successful it correlated to its own payment conversion window opening and not another unrelated payment windown from the same user.

One final consideration with events-based reporting tools like Mixpanel can be a gotcha for to marketplaces… In a marketplace, you always have two parties to think about, the seller and the buyer. In our case, we collected money from tenants (buyers) and pay it to property managers (sellers). We had to send two events for every interaction, one with each user’s id. This allowed us to eventually create reports for both the tenant and the property manager separately.

Well at this point we were happy to have a success rate metric. But our sails weren’t full of wind just yet. We were only halfway and found some of our other metrics also needed to be reviewed. The other main metric we used was Monthly Active Users.

Our old definition used the `created a payment` event to define a user as active in a given month. In other words, if a property manager created a payment they'd only be an active user in the month they created all their payments. Notice in the illustraion how our personal Lucy Landlady is crossed out in October and November?;

We had to redefine what “active” actually was within a month, and there were numerous options; Visited the payments page, Created a new payment, had a payment scheduled already, had a payment collected or had a payment paid out.

We realised the intention to use the product on the property manager side was best to constitute a property manager as “active”. This means having a payment scheduled in a month was the best option. The thinking was if the tenant never paid the scheduled payment, the property manager's definition of “active” isn’t impacted. After all, they had the intention to use the product.

The final definition of active was all unique property managers in a month who `had a payment due` OR `created a payment`. Tracking property managers who created a payment in a month meant we included the property manager who had a booking in the current month but set their first payment for the next month. This is very common considering the booking would always cover the first month's rent and the payments feature worked on all other money except the booking

But this wasn't good enough in the end, further improvement was required.

After a few months, we realised this wasn’t ideal. We found that many property managers that stopped using HA payments never deleted their old payments that were scheduled months into the future. This would result in them being “active” for all those months they had a payment scheduled, when in fact they weren't using the product any longer.

We then changed the definition again from those property managers in a given month who had a payment scheduled to those who had one collected. This we found to be the best overall measurement of product success from the property manager side.

But the problem of property managers stopping using our scheduled payments tool stuck in the spirit of the team. What were we missing? Was it time for a mutiny and a new captain of the pirate ship? Who would walk the plank?

Along with the drop in monthly active users due to the change of the definition of “active” from scheduled to collected, we noted a rather low success rate for payments.

Some of these failed payments we discovered to be attributed to the tenant’s location. Which validated our assumptions that our payment methods available were limiting for tenants. We found that our second biggest market, Germany, was missing SOFORT (a push-based customer-initiated bank transfer). We then added SOFORT to our roadmap.

But an even bigger lesson was on the horizon.

After watching what seemed to be an unusually low number of payment requests being paid, we learned there was something we were missing. How could it be that 1 out of 3 tenants aren't paying their rent on time?

We found a few property managers that had a few consecutive months of unpaid payments. We then lined up some phone calls to try to understand what was going on. What we found was that some property managers stopped using the payments feature, but never cancelled their payments. These payments were sitting there, not dead nor alive to the user, but influencing our reporting each day, month, quarter and year. This was the moment we discovered “Zombie Payments”. Zombie payments are those payments that should have been cancelled, but instead, sit there doing nothing.

Zombie payments lead us to more questions, which helped us verify the position the team was in and helped us make the next moves to steer our ship towards a more successful product.

The main two are;

• What situations lead to property managers having payments that become no longer necessary?
• When in these situations, why aren’t property managers cancelling these no longer required payments?

These two questions would be the next stage of our user research, and by this stage, the metrics had served their purpose. They represented real negative feelings of users and along with other tools, gave us a health check on our payments feature.

With this, we were able to steer the crew to safer waters by improving our product through learning, validation, and discovery.

Fair winds to all ye fellow pirates!