Search This Blog

Monday, 7 July 2025

What does it mean to make a good decision?


We make hundreds of decisions every day, both professionally and in our private life. From deciding to not order that cookie with our morning coffee, to making critical decisions that can impact millions of people for years to come. Would stand to reason that we are all expert decision makers? There is a lot of evidence to the contrary. Here are common traps people fall into:

Judging decision by outcome

Most of us judge how good a decision was by its outcome. Take a moment to think of a good decision you made recently. Chances are, you picked it because it had a good outcome. When things go south, we automatically assume we made a bad decision.

Its folly is hard to see because it usually works. For example, you gained 20 kgs probably because you have been ordering too many cookies with your morning coffee.

However, imagine that you had $1,000 to invest and you see that your local lotto has $100M jackpot. Here is a chance to 100,000x your savings. The bigger the jackpot, more people shell out for tickets. I've spoken to some retirees who play lotto religiously, spend well over $1,000 a month, hoping to turn their life around one day when they win. Well, there is no world where this "investment" makes sense. Even if you win, it was a stupid decision. Because if you made this bet a million times, you would probably not win the jackpot all of those times. The only time you should make this bet is if the lotto is rigged and you have inside information on which numbers to pick.

A good outcome doesn’t make a decision good, and a bad outcome doesn’t mean the decision was bad.

Data driven decision making

Most of us have not built a healthy enough scepticism for numbers. Give us a powerpoint slide with a nice story using a few tables and charts and we would sign our life away. After all, you are making data driven decision. What can go wrong?

Way too many things unfortunately. Was the data used accurate? relevant? complete? Is the analysis done correctly? Is the story told in a way that is not misleading? Are the conclusions drawn from the data correct?

For example, imagine we put together a team to thoroughly analyse the fortune 500 companies. Identify the top 10 attributes that are common among the ultra successful. If we hired a good writer to put together a book on our findings, it is guaranteed to be a bestseller. You'll find plenty of papers in management research literature just like this. We have instagram influencers and marketing firms send out surveys to their followers and publish reports on their data that supports the conclusions they want to promote.

This, unfortunately, could not be any less scientifically rigorous. Even when there is no fudging going on, there is huge selection bias on the data. The data is incomplete. Most likely not representative of the population or reality. If you looked at all the companies instead, you would find lots of example of companies that are not anywhere as successful but have those 10 attributes.

You make decisions on bad data and/or analysis, your decisions will be just as bad. Garbage in, garbage out.

Technology driven decision making

This is similar to data driven but comes from blind trust in technology. Think of people's reliance on GPTs for example over traditional searching. Not that searching online always gives you the right answer. For example, studies have found that these models most often confidently return incorrect information on current affairs. This makes perfect sense as it takes time to re-train these models with the latest data.

We have a blind trust in technology in general. I was once part of a company where the CEO came to me one day to discuss how some machine learning could be integrated into our product. Another team member had "trained a model" to predict when customers would churn or something like that. So I went over to have a chat. After giving plenty of compliments, I asked simply, how accurate the model was. He could not give me an answer. That told me all I needed to know and killed all my enthusiasm. He probably copy pasted some example python code from the internet and tried to fit our data into it. I asked how he validated the model. "What is that?" he replied. If I had no experience with machine learning, I probably would have been jumping up and down to integrate it too like our CEO, waste weeks of my teams time working on something that misleads everyone.

This probably comes from our fear and awe of the unknown. There is this thing that I don't understand. I also don't have the time, energy or ability to understand it. But there is a lot of hype around it. There must be something to it. So when someone tells me everything I want to hear, I am going to trust it blindly, play cheerleader and hope it solves all my problems. Wishful thinking at its finest. However, ignorance is not bliss.

Authority & Incentives

I think there is no avoiding this one. Because of the complexity of the world we live in, we have to rely on experts to help us navigate and make good decisions. But it also does not hurt to apply a bit of healthy scepticism.

To give a personal example, I am losing my hair. There is androgenic alopecia in my family. My wife sees a lot of social media marketing promoting hair clinics says, "why don't you go to one of those." I look at the same clinics and I see mis-aligned incentives, lot of paid reviews and unsatisfied customers saying their treatments do not work. Programmers want to write code, butchers want to cut and hair clinics want to sell hair treatments. There is nothing wrong with that. Before I resign myself to a lifetime of paying for repeated hair loss treatments, shouldn't I at least talk to my doctor first - who is not incentivised to sell me hair treatments?

So when your consultant from the company that specialises in building data lakes says you need a data lake, even if you know nothing, maybe it is a great idea to get a few second opinions from people who has other things to sell?

Conclusion

Making good decision is hard and there is a lot more to be said on this topic.

A good decision isn’t one that just feels right or happens to work out. It’s one made with clear thinking, decent inputs, and an awareness of your blind spots. It means digging deep, checking your assumptions - doing your due diligence. That means not getting too swept up by flashy data, not trusting tech just because it sounds fancy, and definitely not outsourcing your brain to someone with a sales target.

You don’t need to be perfect. You just need to be a little more curious, a little more sceptical, and a little more willing to ask the dumb questions.

Thursday, 7 December 2023

Basics of working with date and time for Software Engineers, Developers and Programmers

Mastering Time in Software



Time is one of the trickiest concepts in software development. Well, it is not. I am just baffled by how many things with software around me get it wrong from IoT devices to big vendor SaaS solutions. If you don’t handle it correctly, things can break in surprising ways. This post will give you a solid understanding of how to think about and work with time in software.

We’ll cover four key topics:

  1. Time Offsets – Why you should always attach offsets to your date-time values.
  2. Time Zones & Daylight Savings – Why time zones are not the same as time offsets.
  3. Unix Timestamps – The binary representation of time.
  4. ISO 8601 Timestamps – The text format for working with time.

I assume you already know how a calendar and a clock work—otherwise, this would be a kids' video, not one for programmers.


1. Time Offsets

A time offset is the number of hours added or subtracted from Coordinated Universal Time (UTC) to get local time.

Take a look at this UTC offset map. Each vertical line represents a time offset boundary. The UK, located around 0° longitude, uses UTC+0, also known as Greenwich Mean Time (GMT) or Zulu Time (Z). Other regions shift their time forward or backward relative to UTC.

Example:

  • Sydney, Australia, is UTC+10. So if the sun rises at 6 AM UTC in Greenwich, it also rises at 6 AM local time in Sydney—but 10 hours earlier in UTC.
  • Some islands in the Pacific use UTC-11, meaning Sydney gets their sunrise nearly a full day before them.

This is why storing local time without an offset is dangerous. If someone in another time zone reads that data, they won’t know what it means without an offset reference.

Best Practices:

  • Always store time offsets when saving date-time values.
  • Or, even better, store everything in UTC and only convert to local time when displaying it.

2. Time Zones vs. Time Offsets

A time zone is a geographical region that follows a set of time rules, including daylight savings time (DST). A time offset is just a fixed numerical adjustment (e.g., UTC+10). These are not the same thing.

If software asks for a time offset (e.g., "Enter your offset: GMT+10") without mentioning a location, I know it’s going to be wrong for half the year.

Example:

  • I live in Australia/Sydney.
  • For half the year, we use AEST (UTC+10).
  • The other half, we use AEDT (UTC+11) due to daylight savings.

Without tracking both the time zone and the offset, your software may display incorrect times when daylight savings kicks in.

Best Practices:

  • Use time zone identifiers (e.g., "Australia/Sydney") instead of just offsets ("+10:00").
  • If using system time, most modern OSes handle daylight savings for you.
  • If working with embedded systems (e.g., drones, microwaves), you may need to manually account for DST changes.

3. Unix Timestamps

Unix timestamps are the most common way to store date-time values in software.

A Unix timestamp represents the number of seconds since January 1, 1970, 00:00:00 UTC.

Key Features:

  • Always in UTC (no offsets needed).
  • A simple integer, making it fast and efficient for storage and calculations.
  • Most databases store date-time values as timestamps under the hood.

Common Variations:

  • Standard Unix timestamp: Number of seconds since epoch.
  • JavaScript timestamp: Number of milliseconds since epoch (Unix timestamp × 1000).

The Y2K38 Problem

  • Unix timestamps are typically stored as 32-bit integers.
  • They will overflow on January 19, 2038.
  • The solution? Upgrade to 64-bit timestamps (which will last billions of years).

4. ISO 8601 – The Best Text Format for Time

If you need to store or transmit date-time as text, use ISO 8601.

Example:
2023-11-28T21:34:57.123Z

  • YYYY-MM-DD (Date)
  • T (Time separator)
  • HH:MM:SS.sss (24-hour format with optional milliseconds)
  • Z (Zulu time, i.e., UTC)

Why ISO 8601?

  • Unambiguous: No confusion between 01/02/03 (is that DD/MM/YY, MM/DD/YY, or YY/MM/DD?).
  • Sorts correctly: Alphabetical sorting works as chronological sorting.
  • Widely supported: Most programming languages and databases support it.

If you must store local time, always include the time offset:

  • 2023-06-28T21:34:57.123+10:00 (Sydney during standard time)
  • 2023-11-28T21:34:57.123+11:00 (Sydney during daylight savings)

Final Thoughts

Time handling in software is tricky, but here are the golden rules:
Store UTC timestamps whenever possible.
If using local time, always include a time offset.
Prefer ISO 8601 for text representation.
Use time zones ("Continent/City") instead of just offsets ("+10:00").
If using Unix timestamps, ensure your system supports 64-bit timestamps before 2038.

Get these right, and you’ll avoid painful time-related bugs. 🚀





Saturday, 11 April 2020

JWT Considered Harmful

selective focus photography of yellow padlock

Recent brush with a program written by a self made JWT evangelist has filled me with rage. Few months ago when we spoke of the same topic, I could not fathom why he was so excited by this technology. After reading the program, I understood that he completely missed the point.

I believe two things need to put as disclaimer everywhere JWT is mentioned:

First, JWT does not encrypt data.

The programmer had to do email validation prior to an unauthenticated action. The programmer received a request with an email address. He then responded to the request with a JWT token and sent a verification code to the email address  The user then had to send a second request with the returned token and the verification code to proceed. The task is simple enough. Except, he returned a JWT signature as his registration token. He put in the entire registration request as well as the code to be verified in the signature.

Why did he do it? My guess is that he did not want to create another database table or use a cache to store the temporary registration data. Or maybe thought he had discovered the new holy grail of data encryption. He also missed the memo: JWT does not encrypt data.

Beside all the ifs and whens of when the JWT secret is compromised, JWT is designed for just one thing: prevent short term data tampering. JWT signed tokens may look like encrypted data. However, it is just three JSON documents, base64 encoded and concatenated together with a period (.). First document specifies which signing algorithm is used. Second document specifies the entire input payload. And the last document contains the signature. It is probably base64 encoded only for the purpose of easy splitting as period is not one of the 64 characters used in this encoding.

This email verification could be easily hacked by base64 decoding the payload. It is a good thing we didn't release this API.

Second: JWT does not remove the need to re-verify requests.

Another horrible way I've seen JWT used is to store partial verification/validation state. A request would be sent. It would be verified and "encrypted" as a token. This token would be returned to the user or put in a data store. Subsequent requests would assume that the signed data valid.

I believe the terms he used was: 'It is cool. It is like how accounting/legal documents are signed.'

Well, it is not cool. What if the request is no longer valid? State of the system could have changed in the time, making the request no longer valid. And of course, the JWT secret could have been compromised - it was just an easy to guess static hard coded value in the code.

I am not saying JWT is useless. It has its uses and it can be used properly for its uses. For example, a  college used it to sign chat messages for a remote medical consultation app. A JWT secret was used during chat session initiation and subsequent messages were sent signed using that secret. I've used it to sign web-hook event notification callbacks to insecure endpoints.

In summary, don't use JWT as the holy grail for security. If the intent is to not expose data, don't send it over the wire. If you have to, use secure transport and/or proper encryption algorithms. Don't rely on it to not need re-validate requests as they may have become invalid.