Blog

In our last post we looked at problems with the inconsistency of configuration file formats, but what's the answer? Most config files have some sort of tree structure – consider a webserver as an example, these have a set of server names and then within each server there’s a set of configuration entries for locations etc. Would some sort of tree of trees be suitable?

With a tree structure, upgrades could easily insert a new node, or have a well-defined ‘tree diff’ of nodes to insert, change or remove, rather than the less robust method of trying to programmatically slice up and insert text into a config file.

You could also have a grammar that defined permitted entries in the tree, and that definition could be used for validation. Add to this per-node/branch permissions and you have a fine grained permissions structure. Documentation could be included in the tree definition which could be extracted to build documentation, rather like Javadoc does for any Java program or Sphinx/Autodoc does for Python. ‘Types’ could be defined for nodes, e.g. enumerated lists that defined all possible options that could be used by configuration checkers, documentation or GUI tools.

Add idempotent APIs to manipulate the trees and with IaaS, configurations could be stored and applied – e.g. at setup, or any point in the life of a server, a tree representing the config to deploy could be used, rather than effectively trying to mechanically edit config files through pattern matching and anchor points, that can’t cope if a config file has deviated from what’s expected.

This approach is not too dissimilar from Microsoft's Windows Registry that provides a tree of config values and files that can act as ‘tree diffs’ that specify nodes to add etc.

Moving on

There are reasons why it became standard to use some sort of text file to hold configuration data. Many were created in an earlier era of computing; before the likes of YAML, JSON and XML had been invented. Also, in some ways we’re just as guilty – when we needed a config for CodeBug Connect for WiFi; we wanted something simple for users, so we went for a text file. In our defence we do provide an API to update that text file and for other configs we used JSON.

But if the configuration of everything in Linux were to be a tree, it would be easy to create web APIs that could manipulate them. There’s a clear need for this when you look at VyOS, pfsense (I know strictly it’s not Linux but BSD based) or similar, which all have complicated logic to generate config files to drive applications and systems such as firewalls, vpns etc. The GUIs are almost completely disconnected too – when a new configuration option is made available to a config file, completely separate logic needs to be implemented to make that option appear in a menu etc..

There’s hope on the way though in that newer features of Linux handle configuration more sensibly. For example, Netplan, which sets up the network, uses a YAML format, and then has ‘renderers’ that apply it. There are also mechanisms, such as cloud-init to allow these configuration files to be applied to new servers as they’re being commissioned.

The Elektra Project is making early steps and we'll be watching closely to see if more applications adopt it.

There are still opportunities to make this work in a standard reusable way and to create standard tools that can find and merge differences in configs, or for graphical editors to provide more flexibility to users.

Wouldn’t it be easier if Linux, its applications and its utils all offered a well thought out API, achieved through a reusable set of libraries and tooling rather than a jumble of text files?

In the age of Infrastructure as a Service (IaaS) most cloud platforms have APIs to update and read the current configurations. Linux in comparison, tends to have an array of text files. While there are tools such as Ansible, Chef, Puppet etc to automate configuration of config files, they often don’t really understand the text files. Instead, they apply mechanical text transformations with complex expressions to match lines or words or make substitutions. We’d argue there’s a number of drawbacks of having lots of text files for config:

A separate format for every program which results in:

1. Each program maintaining its own parser. This isn’t ideal for security, plus other associated tooling needs to be maintained and developed, e.g. means of generating corresponding documentation.
2. Users having to learn multiple syntaxes.
3. Little reuse of external configuration tools, e.g. Ansible has separate modules to handle configuration of each program.
4. No accessible parser for external tools to manipulate the config files – instead tools have to build their own parsers or some hideously complicated regular expression to ‘find and replace’ that hopefully doesn’t have unintended consequences.
5. 'Blunt' version control. At best, version control considers the contents of the text file. There’s no ‘smartness’ that determines what the change actually means – has a setting been updated, or is it just an updated comment?
6. Permissions granularity – permissions are only enforced by the file system on whole text file level

As computer scientists we look for patterns and opportunities where we can reuse code. But many programs are maintaining their own parsers and mechanisms for generating documentation. Users also have multiple syntaxes to contend with; ‘what is the separator for this file, commas, tabs, colons or equal sign?’, ‘What about comments, are they # , or ;, and are they restricted to being the first character of a line’?

When text files are compared without parsing them, it's difficult to know the meaning of what has changed. Consider when Linux packages are upgraded, the current mechanism is to use diff to compare text files and look for the lines that aren’t identical. Diff can’t determine between a harmless comment added to a line or a breaking change. If instead, a standard structure were to be adopted, there could be a standard tool to determine exactly which nodes had changed, and further logic could be built to determine if this resulted in conflicting settings, or if not, merge them.

With text file configurations, there’s no fine grained security. It’s not possible to restrict access to a single configuration setting, all that can be done is limiting read and/or write permissions of the entire file.

In our next post we suggest where we think development is needed to bring configuration files up-to-date.

In a series of blog posts we look at opportunities to improve Linux to make it serve the needs of Enterprise better. With Android being based on Linux, embedded systems like Raspberry Pi and industrial controllers embedded in all manner of routers, ticket machines, etc. it's certainly made its mark on the world. And that’s not even mentioning the majority of servers and cloud instances on the web that run it, nor Microsoft’s move to add WSL allowing Linux to be used at the core of its Desktop operating system. But we think there are some shortcomings related to the computing landscape of the 2020s.

At OpenLX we make extensive use of Linux, from our LoraWAN base stations that collect sensor data, to our data-warehouse and analytics systems in the cloud. We also use Amazon AWS, Google Cloud and Oracle Cloud services and noticed an opportunity for Linux to do things differently. In all projects, we need to manage configuration so we or our users can make updates or ensure consistency across entire fleets of devices and deployments. And for that, using managed cloud services is easier.

Most of the cloud platforms have a similar approach to configuration and management, with role based security and rest APIs to read and write configurations. In contrast, Linux configuration settings tend to be set in text files, with file-based permissions. In the next series of blog posts, we explore if it’s time to reconsider how configuration and security are handled in Linux?

The Internet of Things (IoT) is a term that has been bandied about for some time now, but lots of people still have no idea what it means or, at best, think it has something to do with fridges re-ordering food you're running low on. Actually, this was the answer I got from my girlfriend when I asked her about IoT. She was also rather sceptical about the whole concept, since the fridge wouldn't know her plans for the week (in/out, entertaining, etc.) nor whether she actually wanted any more of that product.

“It could send me a list by text, I suppose,” she suggested doubtfully, “but I wouldn't want it re-ordering stuff I didn't like or buying more milk just before I go on holiday.”

And therein lies part of the problem. People struggle to see how IoT would really work in their lives and many IoT “solutions” seem to be solving problems that don't actually exist. All of which has sadly made it hard for the public (and the press) to take the topic seriously. Furthermore, people think that it must all be very difficult and so they shy away from trying to create IoT solutions themselves or in the classroom.

However, the Internet of Things is so much more than smart fridges (and even they are rather cleverer than their press would suggest). In essence, IoT is when any everyday object has Internet connectivity, which allows it to send and receive data. So, a plant pot that sends you a text when your plant needs watering, or a house alarm that sends you an email when someone enters your home. It's about building useful, connected devices that make everyday life easier.

IoT is expected to just keep growing, with an estimated 25 billion IoT devices to be in use by 2020. So the big question is: where will we find the expertise to design and create all of these new tools, when we don't have enough computer science and engineering graduates to meet existing demand for their skills?

When trying to predict the future, I think it's useful to look at history. If we go back 20 years, only experts created websites. Nowadays, most people create their own, thanks to platforms, such as Wordpress, which allow users to customise their websites to fit their needs perfectly. These platforms were a success, because they made the process much simpler by removing the need to install tools or configure a server; users could simply be guided step by step in their web browser.

If we really want to achieve the IoT vision of smart, connected cities and homes, we need to encourage and empower young people to realise that they can similarly contribute to the Internet of Things today and help shape their digital future. Designing the products for tomorrow doesn't need lots of fancy equipment; you can get started with a CodeBug!

Indeed, at a CodeBug workshop last year, one attendee created a prototype IoT application that warned if a child safety gate was left open - a problem familiar to his family. This powerful example shows how CodeBug brings innovation into the hands of everyone; inspiring a complete beginner to create a device to solve his own need, which could even be commercialised into a new product (with the help of a professional embedded engineer or two).

As with web-developers, there will still be a need for professional engineers and technologists to turn ideas into robust, reliable, well-engineered products. That's why we've created the more powerful, Internet-connected CodeBug Colour - to give secondary school pupils (and others) an easy-to-use device that helps them learn more about coding and electronics and can be used as a basis for developing IoT products. And hopefully, inspire some of the people who will be making 25 billion IoT devices a reality.

Just to finish...shortly after our discussion on IoT, my girlfriend let out an exasperated sigh in the kitchen...she'd run out of washing powder.

“I doubt you can design a 'clever cupboard', which knows what I need, though,” she challenged sweetly, “but maybe you can make a device that I can yell 'need washing powder' at and then when it detects me entering a supermarket, it can yell my reminders back at me, so that I don't forget things.”

Although I probably wouldn't go about designing it quite like that, it seems that there is perhaps room in her life for an IoT solution after all!

If you're interested in learning to create your own IoT devices, why not back our latest kickstarter campaign to get an ESP32-powered CodeBug Colour (with built-in wifi and bluetooth) at a special kickstarter offer price?

Have you decided what you're getting your Dad this Father's Day (Sunday, 19th June in UK, Ireland, US and many other countries)? If you'd like to find a memorable alternative to socks, then you've come to the right place.

Here at OpenLX, we're passionate about quality and simplicity by design and we have a range of fun and easy-to-use devices, which make ideal and inexpensive gifts for gadget-loving Dads this Father's Day. From CodeBug, which is aimed at beginners, to PiFace add-ons for Raspberry Pi, which allow makers to take their creativity further, there's something for every tech-loving Dad.

CodeBug

CodeBug is perfect for gadget-mad Dads who like to tinker with new devices or those who have always wanted to learn to code, but never found the time. A small, programmable and wearable device retailing for just £15, CodeBug is aimed at complete beginners and is so easy to use that anyone can complete their first program in just a few minutes. It's not just for beginners, though, as CodeBug can also be used as the “brain” in more complex computing projects.

You can also personalise your CodeBug gift with ease. Simply follow the tutorial on the CodeBug website to create a basic program with your own scrolling message for your Dad. Download the program to your Dad's CodeBug and it will appear the next time the CodeBug is powered up.

PiFace add-ons for Raspberry Pi

Is your Dad a Raspberry Pi enthusiast? Or does he have a Raspberry Pi languishing in a drawer, because he has run out of ideas on where to take it next? Fear not! The PiFace range makes creating with Raspberry Pi as easy as pie!

Are you spoilt for choice thanks to the many PiFace boards available? If so, we recommend PiFace Control and Display 2, as it will allow your Dad to work easily with inputs and outputs and connect things like motors, switches and lights and it is therefore perfect for a wide variety of fun and interesting projects.