The product was originally written in MEVN (MERN but Vue) stack. We had to solve existing bugs and add new functionalities on top of it. Also, we had a very small team of devs at that time, and the startup was very small, all these factors do add up in the decisions made below.

Some significant highlights of the work:

Reduce DB usage from 92% to <2%:

Problem: We had an API where we had to send push notifications to students when the lecture started. The API ran a for loop for each student where each iteration queried one fcm token, sent a push notification, and moved to the next iteration. This was blocking and too slow.

Solution: I changed the DB query to load all students fcm tokens at once, and send push notifications through fcm batching.

Move website hosting from GoDaddy to Vercel:

The websites were deployed to a GoDaddy server, which had Apache serving all the websites, and the SSL management was manual. Updating sites had cache issues, the site was not being updated for end-users.

So we moved the website hosting to Vercel, we got automated SSL, no servers to check, no cache issues, the Git workflow ✨, and got deploy previews, which has improved the overall velocity from development to production 🚀.

Fix hot paths perf issues:

Problem: We suddenly had our 2 big clients(big by the number of students) conducting lectures at the same time, and our website/apps started to feel slow for everyone. The lag went up to sometimes 5 seconds per request.

Solution: We identified the slow spots first, which were loading and joining lectures. We introduced Redis for caching the lectures, and the keys for joining the online lecture. We instantly saw a performance boost in our server requests, as the round trip to DB was saved.

We also reduced the data sent to the client in APIs which were called regularly, and cached data which changed less often (for example subjects) on the client.

Cache APIs with Redis:

After adding Redis to our lecture APIs, we started migrating more API endpoints to using Redis, and that has brought us a ton of performance improvements, with little effort. This gave us room to solve other bugs in the software, rather than sitting up hours just to solve perf issues.

Implement Backend Server Monitoring:

There were times where we had issues in production and we had no clue what was causing it, and the steps to reproduce did not actually helped everytime. I added Pino logger to log the requests with redacting personal information of clients. This helped us a lot in finding unknown bugs, and even found bugs before the users may encounter them.

Learnings:

- I learned a lot about scaling all parts of the backend, the database, the application servers, load balancers and implemented scaling into the system.

- I understood the importance of monitoring an application for performance and errors, especially error logs, without them you are headed no-where in debugging issues.

- I learned about profiling applications, different tools used for it, and how to step by step profile and solve issues in the application.