Andy posts on The Ongoing Struggle a story that anyone developing an electronic product but especially an Internet of Things (IoT) product should heed.

The subtitle is the NTPmare shortly after Christmas:

Nearly two years ago, on the afternoon of Monday 16th January 2017, I received an interesting BitFolk support ticket from a non-customer. The sender identified themselves as a senior software engineer at NetThings UK Ltd.

85.119.80.232 is actually one of the IP addresses of one of BitFolk’s customer-facing NTP servers. It was also, until a few weeks before this email, part of the NTP Pool project.

“Was” being the important issue here. In late December of 2016 I had withdrawn BitFolk’s NTP servers from the public pool and firewalled them off to non-customers.

Network Time Protocol (NTP) is a means by which a computer can use multiple other computers, often from across the Internet on completely different networks under different administrative control, to accurately determine what the current time is. By using several different computers, a small number of them can be inaccurate or even downright broken or hostile, and still the protocol can detect the “bad” clocks and only take into account the more accurate majority.

NTP is supposed to be used in a hierarchical fashion: A small number of servers have hardware directly attached from which they can very accurately tell the time, e.g. an atomic clock, GPS, etc. Those are called “Stratum 1” servers. A larger number of servers use the stratum 1 servers to set their own time, then serve that time to a much larger population of clients, and so on.

Basically it seemed that NetThings UK Ltd made remote controlled thermostats and lighting controllers for large retail spaces etc. And their devices had one of BitFolk’s IP addresses burnt into them at the factory. And they could not be identified or remotely updated.

Never, never, never hardcode someone else’s Internet address into your code and rely on it 100% with no way to remotely change things. BitFolk was perfectly in their rights to make the IP private for their own security, yet Netthings was using their services, albeit lightly, to rely on their products being able to tell time.

The Company (Nettalk) ceased to trade with effect from 15 November 2018.

So the company went out of business. All their customers now have devices that fail to work.

All should heed this story when developing their products. Read the full story here.