When Data Scientists Met AI Engineers

A few weeks ago, I attended an event called "Future Leaders - Alumni Network" in London and since then I've been left disturbed.

At one point, the discussion was about:

“How exactly will AI impact jobs and society?”

I realised that, despite being able to sketch on a whiteboard how GenAI, RAG, the WINS framework, and Agentic AI all work — and even why CxOs are drawn to these technologies — I couldn’t give a clear, academically grounded answer to that question.

But this week, whilst enjoying the half-term break from day job, I discovered Paul Krugman’s “Hot Dog and Bun” thought experiment.

Krugman’s Hot Dog and Bun Thought Experiment

Imagine an economy that produces only two things: hot dogs and buns. Each hot dog requires exactly one bun. They are perfect complements.

Suppose both sectors employ 100 workers, each producing 100 units a day — 10 000 hot dogs and 10 000 buns. Output is perfectly matched, labour fully employed, and wages equal across both industries.

Now imagine a technological improvement doubles productivity in the hot-dog sector. With the same 100 workers, it can produce 20 000 sausages.

Finish Reading →

First solder in 30 years

It’s nearly 1 month to this date when my daughter and I attended the RC2014 meet up in TNMOC in Bletchley. But it’s only today we chose to assemble the soldering kit and work table to do the start soldering. I had been reminiscing about this in my previous blog post, as a teen, on how I used to etch circuits on copper circuit boards and build a door bell.

This time we are assembling the 8 bit RC2014 computer. But before that I wanted to check if my eyesight can handle soldering. So we decided to practice soldering by assembling a cheap £5 kit - the TTL Logic Probe.

And yes, we did it. Was my daughter’s first ever solder and mine after 30 years. The soldering of the IC looked hard, so gave that piece to my daughter. We did not have 5v battery holder with 2-pin JST, instead we used the 3v battery holder that we got with microbit kit. It worked, and the happiness of having to see something glow after spending nearly 3 hours, is a great one.

Learning by doing is a hard habit to get into. In my previous company we had some really good engineers who we totally into building 3D models, assembling split keyboards. Was fascinating to see them get time to do that back then, but I do realise now that time is available in plenty - we just have to spend it on doing what we love.

Coding with DNA

Today saw the return of me attending the friends of imperial lectures, that I used to never miss till about 4 years ago, since 2010.

The topic was just brilliant - Designing and Writing Synthetic Genomes to Advance Biology

At one point, when the audience asked what Tom Ellis, the professor, was quiet excited about, his obvious reply was the impact of AI and in the elaboration of that he mentioned how he was looking forward to create a prompt, with context and finesse the output of an LLM that produced Genetic Sequence. That was a jaw dropping moment for me.

It was also great to see the old acquaintance of members of friends of imperial, with whom I had visited CERN back in 2010.

Visiting Imperial college always leaves me with some inspiration and excitement about the future.

The Notebook of Data: A 50-Year Journey Behind TRUNCATE and VACUUM in Apache Spark

When researching for my upcoming book on Data Engineering, I decided to trace back the origins of some of the most “taken-for-granted” operations in Apache Spark — things like TRUNCATE and VACUUM.

At first glance, these commands seem mundane — one clears a table, the other cleans up old files. But when you go down the rabbit hole, you realize they are descendants of some of the most profound computer science ideas ever invented.

This post is my attempt to tell that story — through the metaphor of a humble notebook, and the decades of innovation that turned it into the high-performance distributed data engines we use today.

Finish Reading →

AI-102: More Than an Exam — A Map of AI Possibilities

I recently passed the Microsoft AI-102: Designing and Implementing an Azure AI Solution exam — though just by a whisker. It was one of those experiences that remind you how broad the AI landscape has become.

The toughest part wasn’t the theory. It was the sheer number of AI services you’re expected to know — what each one does, how they differ, and when to use them. Add to that a set of coding-style questions that test practical familiarity, and you realise: unless you’ve built and deployed solutions hands-on, the concepts alone won’t carry you through.

If you’re preparing for this exam, here’s what I found matters most:

Do the labs. Reading or watching videos won’t wire the knowledge into muscle memory. Spend time building solutions using Cognitive Services, Language Studio, Azure OpenAI, Custom Vision, and the rest.

Understand the why, not just the what. Every service exists to solve a slightly different problem — recognising when and why to use each is the key.

Expect real-world design questions. These scenarios force you to think like an architect, not a developer ticking boxes.

One side benefit — and perhaps the biggest value for senior engineers and tech leads — is that this exam serves as a catalogue of AI possibilities. You end up seeing the full spectrum of what can be built: from computer vision to conversational AI, from document intelligence to custom embeddings. It sparks new ideas for how to combine services into real solutions.

So yes, AI-102 is challenging. But if you treat it as an exploration, not an assessment, it becomes a creative journey — one that leaves you with both certification and inspiration.

From Data Solutions Architecture to Cybersecurity: A Journey of Continuity, Not Transition

My first foray into Application Security dates back to 2003, when I developed the Authentication and Authorization module for an ASP.NET 1.0 web application used by the State of Maine’s Center for Disease Control & Prevention for Breast and Cervical Health Screenings. That early responsibility—designing secure password storage mechanisms and implementing encryption—instilled in me the fundamental principles of security-by-design. It also gave me an enduring appreciation for the importance of protecting data at its very source.

In the years that followed, across the various public-facing websites I built and maintained, the OWASP Top 10 remained my trusted reference for web application security. Yet, it wasn’t until March 2019 that I became formally embedded within a Cyber Security division of a multinational enterprise. There, I gained a deeper appreciation of cybersecurity at scale—particularly in the context of securing global digital estates.

My team’s focus on Vulnerability Identification exposed me to one of the classic challenges in cybersecurity: asset discovery. Applying my Data Lake Strategy expertise, I helped create a security-tool-agnostic “Know Your Estate” solution, enabling visibility across thousands of assets. By leveraging “heartbeats” from multiple systems, we were able to establish a truthful and dynamic view of active hosts, forming the foundation for measuring coverage of intrusion detection systems such as CrowdStrike, Qualys, and Microsoft Defender.

A security consultant and close friend later told me that such innovation—bridging data architecture with security operations—was rare, as many enterprises focus mainly on compliance. That observation became a turning point, inspiring me to specialize further in cybersecurity while continuing to operate as a Cloud and Data Solution Architect.

To formalize this growing expertise, I pursued and earned a series of Microsoft certifications between 2020 and 2021, including:

  1. Microsoft Azure Architect Technologies
  2. Microsoft Azure Architect Design
  3. Microsoft Azure Security Technologies

From 2022 to 2024, while serving in senior leadership roles, my involvement deepened to encompass the implementation of security controls across the organisation. This extended beyond technical measures such as tenant-level IAM tightening to include process-level initiatives—for example, Cyber Awareness Training for Joiners and Movers and Attack Simulation exercises using Microsoft Defender.

My most comprehensive cybersecurity experience came during my tenure as Chief Technology Officer (CTO), where I led the organisation’s journey toward ISO/IEC 27001 certification. The one-year preparation involved drafting and operationalizing numerous security policies and procedures, embedding core security principles into every layer of the business. This culminated in the company achieving full ISO 27001 certification—a defining milestone that validated both our practices and my understanding of cybersecurity governance.

Having accumulated practical, leadership-level experience in implementing and governing cybersecurity programs, I felt it was time to formalize my expertise through professional certification. This led to my successful completion of the (ISC)² Certified in Cybersecurity (CC) certification—an important step in consolidating my knowledge and aligning it with globally recognized standards.

Whist I am continuing to architect, implement and deliver big-data scale data solutions for AppSec, my goal is to progress toward the CISSP certification, not as a career shift, but as a natural evolution of my ongoing journey—integrating data, cloud, and cybersecurity into a cohesive discipline aimed at securing information systems holistically.

A father’s past and a daughter’s imagination, sparked by 32K of memory

From Bangalore’s hand-etched circuits to RC2014 kits — a journey rewired

On the 20th of September 2025, my daughter and I spent a day at the National Museum of Computing (TNMOC) in Bletchley Park for the RC2014 assembly event. It was, without exaggeration, one of the most special visits I’ve ever made to the museum—and I’ve been there countless times.

A Place That Never Loses Its Charm

My first visit to Bletchley Park was back in 2008. Tickets were just a few pounds then, and I still remember seeing the Colossus machine rebuilt and working fine. Ever since, TNMOC has been inseparable from my visits to Bletchley Park—no trip ever felt complete without stepping into those rooms filled with the hum of history.

There’s a personal thread here too. My parents and sister once stood in the middle of the World War II experience at Bletchley, a powerful reminder of how computing and codebreaking shaped the course of history. Later, during my time at Elastacloud, I visited the museum many times—sometimes for executive meetings, sometimes simply out of that enduring geeky connection.

Back to My Roots in Electronics

Finish Reading →

Two friends on a bench

A lot of thinking happened on the daily walks to this place during Covid-19. Revisiting it on a quiet evening brings back similar thoughts and experiences.

Flirting with Stocks

Finished reading: Flirting with Stocks by Anil Lamba 📚

Two important take aways:

  • To be a successful investor, you need to understand just two things:
    • What to buy (depends on your ability to do fundamental analysis)
    • When to buy and When to sell (depends on your ability to do technical analysis)
  • Read the balance sheet of the company before investing in. Ensure the company is not violating fundamental rules of good finance management. (this is also useful to make career decisions either to join or leave the company)

The DOSE Effect

Finished reading: The DOSE Effect by Tj Power 📚

The four chemicals:

  • Dopamine - The MOTIVATIONAL chemical

  • Oxytocin - The CONNECTION chemical

  • Serotonin - The MOOD & ENERGY chemical

  • Endorphins - The DE-STRESSING chemical

Why India’s IT Talent Is Trapped: A Conversation That Got Me Thinking

two people conversing about difficulties in talent pool while carpooling

Friend: “I can’t find the right people for my team. It’s a niche project—cutting-edge stuff. You’d think engineers would jump at this. But no one’s interested.”

Me: “That’s surprising. Especially in your company. It’s huge. What’s the problem?”

Friend (sighing): “Exactly that. The company. People associate us with routine, not innovation. Even when the project is exciting, the company brand turns them off.”

Me: “Isn’t that fixable? Can’t you directly reach out to candidates and explain the project?”

Friend: “I wish. All hiring is centralised. I’m stuck with standard Job Descriptions. By the time they reach candidates, it’s like Chinese whispers. The uniqueness of the project is lost.”

Me: “Sounds like a brand problem, not a project problem.”

Finish Reading →

How I Avoided Being a Bad Exec by Respecting the Maker’s Schedule

Two individuals are working at a desk, one using a computer with headphones and the other taking notes.

One article I keep going back to — and often ask my teams to read — is Paul Graham’s “Maker’s Schedule, Manager’s Schedule.” It’s not just a great read; it’s a lens through which I view how to build and work within teams.

Whenever I form a new team for a project, I start by requesting everyone to read this article. It’s more than an ice-breaker — it sparks conversations that define how we work together. Even when I’m not leading the team, I’ve often asked my pair programmer to read it, just so we’re aligned in our daily rhythms.

The essence of the article?

Makers need long, uninterrupted blocks of time to get meaningful work done. Managers work in meetings and context switching. Trying to do both simultaneously compromises both roles.

Finish Reading →

This Microbit Didn’t Spark a Tech Career — It Prevented One

Left side: A young person holding a Microbit, Right side: Same person sitting back in reflection, notebook open, perhaps next to a book titled “Business Administration”, Overlay text: “Test the dream before funding it.”

A few weeks ago, my friend from USA reached out to me with a dilemma. He wanted to support a young man’s further education — someone from a small town in South India — but there was a twist. This young person had a Bachelor’s degree in Commerce but was now insistent that his future lay in IT.

My friend was unconvinced. And honestly, I was equally puzzled.

We decided to speak with the young man over a video call to understand what drove this sudden pivot. The call didn’t offer much clarity. All we got was a strong sense of his “passion” for IT — but not much depth on why he felt this way. There was no clear articulation of what aspect of IT excited him, no tangible examples, and no signs of experience or experimentation.

Luckily, a few days later, my friend got the opportunity to meet him in person in Bangalore. When I also happened to be in Bangalore the following week, we compared notes. My friend’s impression hadn’t changed much — he still felt the young man was directionless and was wasting his solid foundation in commerce. He urged him to consider an MBA instead of an MCA (Master’s in Computer Applications).

We were stuck — at a decision point, and the application deadlines for both courses were fast approaching.

That’s when we decided to test the hypothesis.

Finish Reading →

What a Barista Who Couldn’t Speak Taught Me About Leadership

interior decor of Starbucks Kalyan Nagar Bangalore India

I’ve been coming to the same Starbucks in Kalyan Nagar, Bangalore, since the day it opened about three years ago. While many think of coffee shops as noisy or distracting, this space has been my sanctuary for focused work and original thinking. It’s the closest I can get to a non-home environment that allows deep reflection.

Over time, I’ve seen this store evolve like a finely tuned instrument. Operations run like clockwork now—orders are fast, the atmosphere is clean, and the staff are always warm and efficient. But it was only recently that I began noticing something much more profound—something about culture, leadership, and humanity.

Here are three takeaways that made me see this Starbucks not just as a coffee shop, but as a quiet masterclass in leadership.

Finish Reading →

The Grand Anaicut Canal

I had [written](Technology Shapes Society: From Kallanai to AI, A Personal Story www.linkedin.com/pulse/tec…) about how this British built canal transformed and levelled the cultural and economic scene of Kaveri delta region of Thanjavur.

Here is a picture of that canal still being used to that effect and some images from the plantation of paddy for the new season.

Cardiff in Thanjavur!

Was surprised to see this decent cafe in Thanjavur, India.

Cloud Mesh

Time is your enemy

Microsoft Fabric Data Engineer Associate certification : DP-700 exam

Having experienced this exam today, I can easily conclude that the biggest problem is not having enough time to answer all the questions.

Especially, if you are unlucky like me, to get the case study questions at the end of the exam instead of at the beginning - time just flies answering the first section. I was left only with 10 mins for the case study 🙁

So it’s more important to be fast in reading the questions. Most questions take long to read. If your first language isn’t English or you have difficulty reading fast in English, see if the exam is offered in your native language. I know for sure it’s available in Spanish and few other languages.

So don’t hesitate to take up the exam in your language of proficiency.

Good luck!

The Courage to Be Happy

Finished reading: The Courage to Be Happy by Ichiro Kishimi 📚

My fav topics

Life begins from incompleteness Civilisation is a product of the need to compensate for the biological weakness of the human being, and the history of the human race is the history of triumphing over its inferiority.

How many close friends do you have You cannot talk frankly about everything with a normal friend? Why can’t you take off the mask when you are with a “normal” friend.

From an ‘Art of Being Loved’ towards an ‘Art of Loving’ It is difficult to be loved by another person. But loving another person is a task of far greater difficulty

Race vs. Marathon: The Two Leaders I've Chosen

In my career, I’ve had the rare opportunity to experience two vastly different types of leadership. They’ve left such a deep impression that I now see them not just as management styles, but as philosophies of life.

Picture this.

One type of leader treats the workplace like a race.

  • Win fast. Win hard. Win at all costs.
  • Efficiency is everything. Drive the team like an engine on max throttle—results over relationships.
  • It’s about hierarchy, competition, and stories of lone heroes rising above the rest.
  • There’s praise, yes—but it’s often delivered by contrasting one teammate as “brilliant” by calling another “not good enough.” The goal? Inspire through comparison. Motivate through fear of inadequacy.

The other treats work like a marathon.

  • Pacing matters. Shared progress. Winning together.
  • Effectiveness over frenzy. Use the car only when needed—not just because we paid for it.
  • Here, growth isn’t measured just in revenue or headcount, but in how much people actually want to come to work.
  • There are no whispered stories of who’s underperforming. Instead, leaders work hard to make everyone’s strengths visible and valued.

Finish Reading →