Minimal movement of data remains a key need for our customers. So when a customer started seeing unusual issues on an ongoing basis we decided it was time to make a permanent fix. In today’s world of strict IT firewalls, multiple global sites, differing IT platforms and indeed, differing opinions on the right way to do it, there can be many challenges associated with this seemingly simple task. Despite arguably better technologies being available, FTP remains the most common transfer method in use today within the industry. Unfortunately it is riddled with idiosyncrasies.
We recently experienced a case where one customer was having a disproportionate number of failures in data transfers and, on top of that, an offshore subcontractor they were using pretty much couldn’t get any data to upload reliably. Yet when we tested the accounts from multiple global locations they always worked fine for us. I’m sure anybody who has been in any technology industry for a while will have come across similar types of “works for me, doesn’t work for them” problems. It can be a strain on professional and commercial relationships and such problems can be really hard to pin down.
There were three main things we did to fix this problem:
- Reduced the number of variables: Each user or their IT department had their own preferred FTP client and of course each thought their choice was the best possible client
- Gathered more useful data on the issue
- Documented the tools properly.
To reduce the number of variables we removed any FTP clients, in the conventional sense, completely. Instead we implemented scripts for Linux and Windows systems that utilise the built-in FTP command line tools and native scripting languages on each: Cmd for Windows and Bash for Linux. This reduces the system requirements to the lowest common denominator and removes any dependency on tools not provided by the OS distribution provider. Our scripts are fully open for inspection by IT departments. Using no third party client and having open scripts makes it easier to get IT buy-in.
We were careful to create log files that were easy to understand and formatted for easy review with nothing more than a text editor. Log files for each script run are also themselves uploaded to the FTP server. This is the last task performed. So now if there is an issue the server support personnel can match up the server side logs and the client side logs and see what was happening at both ends of the link, not just the server side. If the link breaks completely the customer can still email the client-side logs to us.
By creating scripts ourselves that are focused on the needs of our customers it was possible to create a single harmonized tool and workflow that covers all but the rarest use cases. Documenting the tool in detail and having the documentation reviewed by the right person enables the tool-set to be used by small companies with no dedicated IT support function or by larger organzations with strict IT policies. Because of this our customers can now benefit from reliable data upload with minimal effort.
These scripts and the documentation are available to all our customers free of charge.