Before the main content: I am creating an R Community on Google Groups. You can join the group using this form.

With Obama’s recent comments on AI, I was thinking about what once was the Open Government initiative that, while remarkable in many senses, used to confuse data and information. That was back in 2009, and a large fraction of my work now and then consisted in obtaining data, this is values might pull out of a database such as region, city and variables of interest provided in formats meant to provide interpretations and drawing conclusions. In other words, a part of my work has the complication of dealing with data provided in the wrong format for such goal, as it is the case of releasing data in PDF format.

When an economist like me goes to a government website such as the USITC, it is often the case that he/she is looking for raw data so that it can be analyzed and shaped in different ways, usually detecting inconsistencies, and interpreted into the form ot summary tables and plots inside a document. USITC uses the correct formats for that goal but other public offices tend to insist on the wrong formats even now that AI has become a salient issue and it is often perceived as a multi-purpose tool.

Sharing data in PDF format will continue to be a common practise, and often once that overly complicates scientific work and investigative journalism as it was the case of Dollars for Docs. The popular saying is that if you have a hammer, then everything looks like a nail. AI-based workflow may increase the speed at which we can create a software prototype and still work within an organizational culture that confuses data and information.

With the rise of AI, government agencies can streamline the workflows that result in a PDF for effective information distribution. The information in a PDF document can be read in its electronic form, printed and reshared easily. However, adding AI to government workflows can add a vicious instead of a virtuous element. For the economist who wants raw data, PDFs made faster as in more frequent updates is the incorrect choice. The risk is that AI can become a new hammer, a tool to try to hit everything with in the same way as we continue to use PDFs without thinking of the suitability behind the technological choice.

I am quite sure that many data issues, not just in government but also in businesses, do not require AI at all. Around five years ago, a much less salient hype was about Machine Learning (ML), and many comments gave me the impression that ML was becoming a new hammer. In particular, many data issues such as bottlenecks and organization can be solved by moving data from multiple files into a proper Structured Query Language (SQL) database engine such as PostgreSQL or MariaDB, both with a proven track record.

Raw data also requires metadata describing the time range, source, and descriptions for the individual variables, and multiple important properties. A PDF seems to be useful to provide a data dictionary, and so could be a Word document or even a plain TXT files, which had led me to think about the lack of debate on the suitability of AI and many other tools. For the SQL case, it is justified for large raw datasets where Excel spreadsheets fall short, and there will be cases where Excel spreadsheet will be just right.

There are many other raw data formats that might be more suited to particular needs such as plain spreadsheet files (CSV). There are specific needs, as it is the case of geographical data, for which we have shapefiles (SHP) for holding spatial feature information with their respective database version known as PostGIS. Like different data format serve different goals, it is up to us to see the AI hammer in the toolbox and know when to use it.

How is it relevant to the R community? I have received an increasing number of low effort Pull Requests (PRs) to the different R packages that I maintain and I see the IA hammer in action. Most of those PRs failed to answer simple questions from my side such as “is this data free to be re-distributed?”. I usually get a positive feeling when I receive an email or see an open issue on GitHub from a user asking me about a particular dataset. A few times, fortunately a few ones, I have received a PR with a simple change because I documented a function with a bad wording or I had typos, and so I am thankful for those corrections that sometimes came without much debate.

I think that I will continue to receive AI-based PRs, which it not bad by itself, and so it is not using it to add more examples, add checks or rewrite parts of a code. My same critique will apply if, instead of AI-based changes I would be receiving changes consisting in copying and pasting without much consideration about the relevance of doing it.