A chapter of my work life comes to a close

I dont know how but I somehow missed publishing this post in time, but better late than never I guess ;P

I recently resigned from Near.co as Technical Architect after nearly 3 years of being with the team. I must give it my leadership there who proactively made sure I was relieved of my duties in a week once they realized I wont budge. Instead of me wasting endless 3 months, doing nothing but killing time, which many companies have, I was quickly transitioned out. I am sure not many folks have had this kind of experience yet in India.

As a philosophy in my work life, I had decided a long time back that I wont resign in haste. But once I resign I wont take it back. This has been how I have been working for past 11 years. In fact I thought about moving out Near nearly 5 months back, but was not really sure about that. But the following months made my resolve stronger. Well I wont go into details why I really quit, because that gets boring :D . But what I would really want to recollect here are some of the amazing experience that I have had at Near, formerly known as AdNear.

So lets start from when I had met @mixdev at Barcamp Bangalore. He was telling us war stories of he is able to scale things  well at AdNear with PHP-FPM and Redis. I was truly impressed by the super low latency he was able to extract for adsearch. For those who dont know him, he was one of the planners for Barcamp Bangalore and I have great respect for him. The building block of tech stack at Barcamp Bangalore were done by him. Infact for the first few years he was solely responsible for maintaining the website, design to dev to devops. I was the guy who pitched in wherever help was needed. So as u can understand now, my great respect for him and when he offered me to work with him at AdNear, it was difficult  to say no to it. I was in a cushy job at Intuit then, working on SnapTax. It was the early avatar of  TurboTax for mobile and I was working on Android version specifically. I was one of the early team members and it great fun working with the team there but three years on everything was getting more or less predictable and I was in real comfort zone there. Guess I was getting too comfy there and decided to take up the offer at AdNear where Arun aka Mixdev was the VP Engineering and  I would work directly with him as Technical Architect. My last day at Intuit was extended by another week beyond my notice period of 1 month, because I wanted to make sure that lean team there did not feel too much stretched because I left them in the middle of  the release with two hands short. I made sure they were in a position of meeting the release dates without too much trouble.

Enter AdNear in November of 2013. I was asked to take charge of the mobile SDK team. The base SDK was in place but then we rallied and added a bunch of features like MRAID and VAST based ads compliance. Also built a bunch of apps which helped the Ad Operations team to test and verify the ads before starting the campaigns.

I was also pulled in to implement the ID + Profile data management system for AdNear's effort to build and use profiles for targeted ads. It was something completely different for me. Understanding the distributed nature of the information flow and keeping ids in sync across different data centers and delivering profile information for more than 500M users(that a was the target then, which we crossed faster than anticipated) and keeping in mind future influx of users as we scale our systems. This was the time when I tried so many different technologies, read and experimented with them to finalise the solution which we deployed. The cool thing of this whole implementation was how quickly it was done, thanks to the highly talented team which was there backing me up.

Another thing I realized during this task was it had become incredibly difficult to look at the logs. Thats when I thought of using Elastic Search and it became obvious that we needed Apache Kafka for it work at scale of things we were handling. It was really difficult to setup the entire system and after learning a lot of details about ElasticSearch I was able to get the systems to work well.

Next challenge placed for me was to scale the Real Time Bidder which was a monolith at that time. We were extracting the best possible performance out of the systems. Next step was to enable horizontal scaling. So we finally broke the entire system into subsystems and made it almost infinitely horizontally scalable. Also we wanted to make deployment simpler, because it used to take just too much time to put new systems in production, so I zeroed in on Fabric after evaluating a few of the solutions.

Next challenge for me was to re-architect the profile delivery system for the custom profiling system that AdNear was building for its new product called Allspark. This meant I would have to search for a new database which would work well with the scale of things that were about to come. Probably 1000s of segments for more than a billion users. This meant using Redis would be costing too much money as all the data would need to be in memory for Redis. So I tried many databases like SSDB, MongoDB with LevelDB and several others. Finally found this obscure database called Kyoto Tycoon which seemed the best thing to be used. It was highly scalable and low latency for read and writes. It was also neat at providing good replication with master slave configuration across data centers.

I also experimented with Apache Storm and found that most suitable for continuous data processing that we were trying to do at AdNear. Further used it to build the profile delivery system which could deliver this data to all the different partners we connected to in a streaming fashion. It was fun to architect the system as it had a lot of unknown and failure points. After a good deal of experimenting, I decided on making the solution simple using Apache Kafka and Apache Storm with a good deal of monitoring systems in place. Made a template solution to connect with different partners as per their API design which made it easy to implement for all the different partners. I also used Apache Storm to build the analytics systems for measure clicks and impressions so that ads which had met their goals could be stopped over the entire geographically spread out data centers.

The other project in which I had a lot of fun implementing was the framework to enable ad delivery and performance optimisation using algorithms that could be plugged in to do their magic. It was fun designing the entire pipeline which depended on other systems for data and once its gather all information it needed, it updated other systems which made improved ad delivery and performance.

During the last few weeks at Near I was experimenting with ScyllaDB which was C++ clone of Cassandra. Its website gives very impressive performance benchmarks. I was trying to find get this up and be able to handle the profile information in a cost effective manner. Also tried Apache Mesos with Singularity so that jobs could be fired using APIs instead of a developer manually starting it on machines.

Also during this time I had the privilege of writing Apache Kafka Cookbook and reviewing two of them, Learning Apache Kafka and Mastering Redis.

So for me its been more or less joyride. I have already decided what I am going to do next and I hope my next chapter of life would be even more fun to look back at this past 3 years have been. But I am not sure if ever in my life I would again get a chance to build systems ground up and experimenting with so many niche and unknown technologies.