Customer Story: Rain (Tech Lead)

Feature Flags for Meeting Customer Needs: A/B Testing, Segmenting, and Kill Switches

We sat down with Dario Boverio, Rain’s Technical Lead. Rain is a Fintech platform that employers use to give employees earlier access to wages.

Rain uses feature flags to segment and personalise the way they release to customers. They can choose which users to roll features out to, run A/B tests for popups to find the best option for users, and implement kill switches to quickly roll back any releases that degrade performance. All of this lets them quickly adapt to changing customer needs.  

This is a two-part series:

• We talked to Dario for a deep dive into the technical implementation of feature flags
• Venu, their VP of Platform Engineering, gave us a high-level look at how feature flags have changed releases for their teams. Find the interview with Venu here.

What’s Rain’s main use case for feature flags?

We have different use cases, and the main use case is around segmentation and feature toggling. Specifically, we use flags when we need to enable a feature for specific employees.

Our final customers are employees, and sometimes we create a feature that we don’t want to enable for every employee at the same time. To manage this, we’ll create segments in Flagsmith and start enabling the feature employer by employer.

Most of the time we will segment releases to specific employers or organizations. Sometimes, though, we release to employees who are in a specific phase of the flow (for example). To do this, we use Flagsmith traits, and we’ve built event-driven architecture to update traits in Flagsmith. Every time a user follows the normal flow in the app (onboard, sign up, enter their address, etc.) we update their traits in Flagsmith.

Then, in every application (and also on the app) we can use this information to create flags with segmentation to apply a feature only for the identity we choose.

Security is often a big piece of the puzzle in Fintech development processes. Does Rain use feature flags for kill switches etc.?

We use kill switches for safety and security when we release. Sometimes we create new, important features in the app and in the backend of our applications. For security reasons, in those cases, we always create a kill switch flag that enables us to stop the feature instantly.

For example, we built a rewards process where we can give benefits to users after things like completing their first transaction. To launch this, we had to add an implementation to one of the most important flows for Rain, which is the transaction flow. If this implementation didn’t work as expected—say it resulted in a performance degradation—we needed to make sure the transaction flow would not be affected.

To handle releases like this we always use a feature flag. That way, if we release and we do see a metric change or a degradation in performance (and we know that it’s because of the new feature) we can turn the feature off in Flagsmith and everything is back to working as it was before, in real time.

Can you tell me about the A/B tests you run?

We run A/B tests for the app (front end) and for the back end. We might test releasing to different employers or releasing different information for specific employees, for example. We have a flow for users and we can decide to run a test and only release a feature to certain users depending on their sign-up behaviour within that flow.  

We implement the A/B testing on our own but using Flagsmith segmentations with a % Segment Override. Then, on our side, we built an event pipeline so that we can send events through the app to the back end to track users’ behaviour in each scenario. Finally, we store the event information in our data warehouse (using Redshift Database).

Then, we use different analytics tools to measure the results of the experiment, like Grafana or Looker (say, if the test is more relevant for Marketing, Product Managers, etc.) and build dashboards so that the right people have access and can make data-driven decisions.

Once we have enough data,  we can definitively certify which scenario converts better based on the data and we directly enable the winner in Flagsmith for all of the users. As a last step, it’s just a matter of removing the old code from the other approach. And that’s it!

Flagsmith makes it easy to implement and change the percentage of users for each segment in real-time.

Do you have a specific use case for the A/B tests?

We had one specific use case where we were not receiving feedback from users because they were closing the modal we had implemented. I think it was because we were requesting a review at the wrong time for users. So we created two different modals in different places in the app. Then, we evaluated how the user feedback varied depending on the modal that they saw.

What’s interesting is that even though we sent the two modals in Slack and all of us internally felt confident about which option was going to win, it didn’t play out the way we thought. Option A was so intuitive for us, but then when we ran the test we saw that option B was by far the better approach for users.

After this new implementation, we are now receiving more feedback than before, and the reviews are much better after the implementation of that option (A).

Table: Dario’s breakdown of releasing before vs. after implementing Flagsmith

What has changed with your development cycles since using Flagsmith?

The main thing is our security with releases. We can wrap something in a flag, turn it on or off easily, and we don’t need to worry. The benefit is also that we can build something and deliver it to the product team to manage everything in the Flagsmith UI, and we don’t need to do anything else.  

The product team can request a feature release. Engineers can agree to turn it on for just one employer. Then we just delegate it back to the product team and they can enable it for more employers as they see fit (depending on business decisions) without the need to ask the engineers. Everything is implemented and they can choose to enable, segment with traits, or set up whatever configuration they want in Flagsmith on their side.

In terms of development and deployment, it’s simple now. And much simpler than having an in-house solution where we need to plug things into a database or configuration files for the product team. This would mean we would need to run a query on the database in production, which is not good, or we would need to change the configuration file for adding the new employer and restart the service.

Everything is much faster now. And the engineers don’t need to support the product team as much, because they can do the majority of it on their own.

How do you deploy Flagsmith?

For our web and mobile app we use the Flagsmith Javascript Client. On the server side, we use the Flagsmith Go SDK.

We also created a wrapper on our side. It’s a library where we put everything related to Flagsmith. All of our services use this library, which uses the SDK to call your endpoint.

Then for the ones that are not included in the SDK, we created a REST API client on our side. So we are using three things: the SDK on the web app, the Golang SDK on the server side, and the REST calls.

That makes sense. Then what do you do for a database on the backend?

We mainly use Postgres, DynamoDB, and Redshift. We also built an AWS Lambda function as a webhook to receive events when a flag is evaluated. Every time we call “Get Identity Flag” for a specific user, we can get the state of the flag for that user and send this information to our data warehouse.

This lets us cross-reference the information with, for instance, A/B tests we run to get metrics and see the conversion rate on specific use cases (for instance, we can add a new view for a specific segment of users in the signup flow and measure whether this change impacts positively on the conversion rate of users registered in the application.)

Do you use any of the integrations?

Not at the moment, but I read a story about another client of yours who uses the Slack integration and that sounds useful for us. It would be helpful to get notified if someone changes a flag value, which could be critical for us. It would also be helpful for security reasons, because there are many people using Flagsmith on our side but not all of them should be changing all of the flags and it gets hard to manage all the security layers. If I give access to someone and they pass access to someone else, it can be hard to control.

We need strong visibility into what people are doing in Flagsmith. Slack notifications could really add to that and help us track what’s happening with our flags.

Is there anything else that you think would like to mention?

Support from Flagsmith has always been there. Every time I need support from them, someone replies instantly.

Thanks so much for sharing your insights, Dario! We really appreciate your insights and time.

To read more about how Rain uses feature flags in their development processes, read the interview with Venu here.