At this point, no one has avoided hearing about new AI-based technologies to the point of exhaustion or bursting. Despite this, it may still not be clear to most how such technologies actually operate and how exactly they should be used. The field of documentation lies at an intersection of many such opportunities and potential threats. As such, I asked for his opinions from a friend of mine who works on an AI-based product. I will also discuss my own observations and thoughts on how AIs relate to work in documentation.
Interview with a professional in AI
This interview was conducted via email between March 7th and 11th 2024. I know Oliver through a shared hobby from the time when both of us studied at the University of Reading. Of course, I translated his replies in the Finnish version of this piece. His original replies are used in the English version.

Interviewee: Oliver Back
Customer support and community manager at Scholarcy
Originally in academia studying liquid droplet robotics, I now work in EdTech with the aim to increase the accessibility of academic text for everyone.
Interviewer: Sami Rissanen
Documentation specialist at DoX Systems
Alleged to once have studied philosophy. No live witnesses could be found for a testimony to verify this. D̵͖̲̹̹̪̼͖̈́̂͑̒̄̏̐͂̎͛̕̚͝ơ̷̧̛͈̩̙̤̭̻̦̣͌̆̓̏̉̀̌ ̸̢̛̞̜̥̱̬̹̥̭̗̻͂͊̂̓̀̓̿̕n̴̨̨̞̥̝̪̦̦̖̱̺̙͙̯̐̈́̈̒̎̿̀̇̎͠o̶̰̥͎̭̗͌̎́t̸̢̹̘̪̟̩̰́͒͒̀̈́͘ ̵̥͖͍̏̿͋͂͐̉̄͐̐͝l̸̖̝̜̣͎̲̝̮̗̱͉͚̀͊͌́͋͐̎̉͝ȍ̵͇̩̠͍̯̺̠̪̜̅́̀̄̒̈́͊̂͒͗͑̀̔͜ǒ̷̡̧̨̩͖͍̳̭͖̥͓̳̭̋͆̍̐͠ͅk̴̻͍̯͙̞͈͎͊͛̎̈͐̅̀̀̀̚̕͝ ̴̮̲͍̬̖̭̟͔̮̿̋̈̀͂͛̾̄̎͛i̸̢͇̗͇͇͔͙͖͍̺̍͜͝n̵̨̥̺͎̜̼̭̦̥̘͎͇̂t̸̢͕͓̣̱͔͈̱̱̖͙̹̄̎́͒̾̿̀̇̆͘͝ǫ̷̼͈̱͚̜̈̆͑͑̍͝ ̶̲͇́͗͑̂͒̏̅̈͒̆͐̕̕͠͝t̷̨̟̮̤̹͈̣̠̃̈͋̾́̈́̕̚h̷̟͙̳̟̜̤͗̃̐ͅî̷͎̹̂̓͆̐͘s̸̢̺̣̳̥̲͔̟̏͘.̸̡̘̪̪̠̺̮͖̘͙̲̻̖͈̜̆̈́̐̒̈́̀́̅͗́̇͋̀͊̚ Otherwise, this one is a mostly harmless stylesheet pusher.

Sami, question 1:
Could you briefly introduce yourself and explain what you do in relation to generative AI?
Oliver, answer 1:
My name is Oliver Back, and I work in the EdTech industry for a company called Scholarcy. I originally studied for a PhD in Cybernetics, where I researched liquid droplet robotics using the Belousov Zhabotisnky reaction. I am an avid gym goer, and my current hyperfocus is Warhammer 40,000.
Scholarcy turns academic articles and textbooks into interactive flashcards which students and researchers can use to screen, skim read, and critically evaluate academic text. Scholarcy uses non-generative LLMs to extract key information, facts, and contributions that authors detail in their articles. These text snippets are then reformatted to increase the accessibility for readers short on time, or less experienced with reading academic text.
Scholarcy uses these key facts, and finely tuned prompts to generate the Synopsis section. This is the only flashcard section which uses generative AI, and is designed to give the reader an overview of the article. This can allow readers to preempt key concepts which come up, which helps readers approach reading longer texts more effectively.
Sami, question 2:
I can see the value in that. Most of the audience for this is likely to work in technical documentation, specifically. Do you have personal experience on which to draw dealing with documentation processes or their products to give context for applications of generative AI there?
Oliver, answer 2:
I think generative AI is a fantastic tool for technical writers to use. But that’s all it is, it is a tool. At this stage it will never replace the human in the loop, as there are so many ways that generative AI can misinterpret things, or not be aware of a specific nuance that the end user requires. In my experience at Scholarcy, I do not tend to use generative AI for any documentation, or user guides that I create, as I feel I know the software better than an AI currently does, and it would take longer to rework that writing so that our users can take advantage of it, without having to come back with questions.
The use of generative AI has come up a lot at Scholarcy, and I happened to have interviewed a cloud applications programmer on their thoughts on using generative AI for both programming, and documenting their work. They felt that it can alleviate some of the pain points that a lot of programmers face, but it comes with drawbacks. No one knows where the data you feed into those models ends up, and no one wants to be the developer that feeds production code into an open system. This creates a new problem, where company specific issues are translated and masked so that those tools can be utilised. The output still needs to be checked, in the same way that copying code from Stack Overflow won’t necessarily work immediately, a developer still needs to understand how to program to take advantage of these tools.
I do think generative AI is, and will be a fantastic tool for technical writers to utilise in their workflow, as it can provide a framework, or skeleton which they can use to fast track their final result, but this is a new skill in itself. As much as copywriting, proofreading, and editing are skill sets in their own right, interpreting generative AI’s writing is a modern take on the established writing process.
If a copywriter or technical writer wishes to use generative AI, they should! It can help skip the slow parts. Every writer understands that sometimes they will be stuck staring at a blank page, wondering how to get started. AI can help solve this problem, it will at least give you some writing to work with.
I’m a big fan of the FBR method for writing. Fast, Bad, wRong. It’s a way to get your thoughts onto the page, before making edits, fleshing out any roughly constructed sentences, and polishing off any explanations. Generative AI can be used in exactly the same way, except rather than spending half an hour writing down all your ideas in the order they pop into your head, you spend 30 seconds writing a brief, add a couple of prompts, and then you’re onto editing and fact checking.
Sami, question 3:
I can certainly see those applications. However, people who work in technical documentation also include people like content architects, technical illustrators, engineers in review duty, and translators in addition to technical writers themselves. Would you have any recommendations for how they could incorporate generative AI to help with their tasks?
Oliver, answer 3:
Generative AI can help to start off the writing process. When there are so many people collaborating on a project it can be easy to find a starting point, as there are so many different insights to include. However, the information overload can present a challenge to getting started. How can one go from a blank page, to a perfectly crafted technical document?
Generative AI can be utilised by writers trying to get started. A brief description of the document, and some carefully crafted prompts can give you a skeleton to work with. Fact checking is a must, along with expanding on details, and polishing off explanations. But using this as a template can increase efficiency, and drop the barrier to entry for many writers.
Another use of Generative AI that may help these types of groups, are meeting recording tools. Having a transcript and summary of a meeting is a fantastic accessibility tool for many, and gives anyone a second chance to catch an important point if they happened to be distracted, or if they are tired or ill.
Sami, question 4:
I can see the potential there. However, as an insider, what would you say are the greatest limitations of generative AI technology either fundamentally or only contemporarily, and how do they manifest themselves?
Oliver, answer 4:
A very interesting question! The possibility for genAI to hallucinate causes a lot of concern that factual inaccuracies will be included within its writing. This can be combated with a proofreader checking through the content, but this relies on their subject matter knowledge being good enough to catch any inaccuracies, or bias that the AI introduces. It can be easier to catch these flaws when there is a smaller volume of writing. But in cases where much longer documentation is required, the time costs and energy to check the entire document can act as a barrier to entry, allowing smaller inaccuracies to slip through the cracks.
Sami, question 5:
And with this in mind, there remains active discussion on whether generative AI technologies are largely a fad with niche long term utility or a true technological revolution. Which avenues for the use of gen AI do you see as having the most potential for long term utility overall?
Oliver, answer 5:
I certainly don’t think that AI is a fad, I don’t think it’s going anywhere. I think there are a lot of varied avenues which will see a revolution with the advent of AI technology. Big data solutions will continue to see innovation, the algorithms used will get bigger, better, and faster. I hope this means that healthcare, and medical research can see some tremendous benefit, but I can also see marketing and targeted advertisements becoming more effective!
I want to see tools used in education that can use AI for good. There are tools on the market already, Scholarcy being one of them, and it is interesting to see how the technology evolves and improves over time. Most students stop developing their study skills once they leave secondary school, so it’s exciting to see how tools like Luna are scaffolding around existing study skills, whilst using AI to point its users in the right direction when it comes to revising the right content.
There is a big push to use AI in trading algorithms, this isn’t new. At university when I was taught the recursive least squares algorithm, the lecturer commented on how it can’t be used to predict stock prices, since the stock market is a time-invariant system. It sounds like a really big advantage, having an algorithm based off prior knowledge of human interactions, market trends, the news cycle, or everything really. But this doesn’t necessarily work for predicting human nature, or even human needs. LLMs work because language is somewhat predictable. New words and grammars can be invented, sure, but this isn’t a fast process, and a lot of new slang follows predictable rules which an AI can adopt and learn. But human nature cannot necessarily be trained into a machine in the same way, so I am not completely sold on AI ‘solving’ the stock market.
I believe the biggest avenue for genAI lies in digital content creation. Writing, editing, and proofreading are some of the most ‘foolproof’ AI technologies on the market, and will only get better. I think it’s likely however that the practical use will entirely change. There will be a point where you no longer need a technical manual, as you’ll log into a website, and use a companies chatbot to help diagnose an issue with a product you bought, rather than reading a pamphlet which it was delivered with, or even trawling through sites online to find someone else that had solved a problem you currently are experiencing. Most new phones and computers are moving away from the set up process you would previously walk through. Most new devices pretty much work from the moment you power them on. Sure there is a set up procedure, but it’s largely just signing into your email, accessing a wifi network, and letting the machine do the rest.
Sami, question 6:
Thank you very much. Do you have any closing statements to cover an angle that the questions did not address?
Oliver, answer 6:
I think the state of AI is interesting, there are so many fantastic practical applications of the technology which can enable so many amazing things. But there is also the hype around the buzz word, where everything has to be AI. A friend even mentioned an AI toaster recently!
It would be fantastic for most people to be able to use AI in their jobs, in their hobbies, or just generally to make their life easier.
I appreciate the questions you have asked, Sami, thank you for hearing my thoughts!
Interview summary
Accessibility lies at the core of Oliver’s stance on the benefits of AI. Such tools can help remove obstacles related to special needs for these tasks. Examples include summaries and transitions between formats such as speech and text. They also act as aids for generating frames on specific topics, as it is easier to edit initial content than to compose new content from nothing.
According to him, the most obvious limitations are related to how unreliable the outputs of AIs can be. As such, it is always necessary that such outputs are not used in their initial state. They must be reviewed and edited by subject-specific experts. As such, you should keep portions generated by AI short before you process them like this.
You should also remember how other forms of AI besides generative AI remain useful to apply alongside it. They can either be supported by it or support it in places. However, in his estimation, the use of this technology will spread over time and it will replace prior operating models.
My musings on AI
Overall, I lean towards skepticism when it comes to generative AI. This is in part why I wanted to also include another’s perspective. Below, I will discuss my own thoughts in relation to both the most fertile use cases for AI in documentation and its core limitations together with its other issues. Any reference here to AI denotes generative AI unless I specify otherwise.
Operating model of generative AI
I find that too many discussions of generative AI do not first actually look into how they function. I am by no means an expert on this but I will provide a summary of my current understanding as context for the following discussion. This summary is especially relevant to my claims about the limitations of such technology.
My interpretation is mostly based on these sources:
- https://www.techtarget.com/searchenterpriseai/definition/generative-AI
- https://machinelearningmastery.com/what-are-generative-adversarial-networks-gans/
- https://towardsdatascience.com/understanding-generative-adversarial-networks-gans-cd6e4651a29
- https://towardsdatascience.com/understanding-variational-autoencoders-vaes-f70510919f73
Neural networks
Generative AI is still fundamentally founded in neural networks. A neural network is an algorithm which consists of
- nodes with values,
- associative relations between nodes,
- tables which connect nodes’ values to possible inputs and outputs, and
- feedback channels.
Nodes are independent data points which may form networks of relations to other nodes. Each node has an identifier value which distinguishes it from the rest and only corresponds to it in a given network.
Associative relations between nodes are weighings between them which represent, for example, the probability that one becoming active also activates the other, or the distribution of changes in weight values between them. They create pathways for the activation patterns responsible for outputs whenever the neural network is fed different inputs.
Tables can then be used to give the states of nodes or sets of nodes meanings which correspond to real world inputs and outputs. Examples include ‘the user wrote “dog”‘ or ‘send an error message’. Such interpreters need not literally be tables but nodes’ states by themselves only become meaningful through such a constructed or generated relationship to reality. They determine which nodes a given input activates and which outputs a given state of the neural network produces when it is recorded.
Feedback channels, on the other hand, allow the networks to adjust to associative relations between nodes by re-weighting them. This requires a positive or negative evalutation given to the system’s output as a means to strengthten or weaken the system’s tendencies until parts of it approach ideal balance states relative to available inputs.
As a result of these components and the operating principles that emerge from them, a neural network can modify itself towards the goals communicated by its feedback system in an iterative and autonomous manner.
Adversarial networks
One form of feedback used by generative AI are generative adversarial networks (GAN). They are not the only such option, and the different methods might be compatible to use together in different ways. Another such option would be variational autoencoders (VAE). They operate based on the gaps open to interpretation left by compression during changes between formats and reconstructing that compressed content alongside filters to differentiate between defective and functional outputs. Adversarial networks are easier for me to describe as an illustration of related principles, though.
Such feedback systems which enable generative networks allow relatively freeform outputs to fit their filtering criteria. In other words, an ideal balance is not restricted to a tightrope. Instead, it consists of a wider field of possible outputs which are compatible with the provided criteria. Because outputs are variable but still tightly controlled relative to fully random alternatives, they become generative. As such, they are creative in the sense that they can represent the intended things without consisting of directly identifiable parts of the used samples.
In the case of a generative adversarial network, the AI is implemented through the use of two neural networks. One has been configured to be a discriminator and the other a generator. The feedback loop between them rewards successfully identifying generated outputs or deceiving the discriminator. The discriminator can only access the sample constituted by its training set but it must determine whether an output fed to it is a part of the same larger set as that sample. When its success rate at identifying the outputs of a generator reaches 50%, the outputs from the generator can be treated as good enough to force the discriminator to make a blind guess. At that point, it thus cannot differentiate between imitation and a real sample. The use of better discriminators to train generators thus also results in generators which are better at deception. Such deceptive generators are the true source of an AI’s creations.
Attention
Another enabler of current forms of generative AI is the attention span built into them. This is their ability to utilize a larger preceding total as part of processing a new input. You can conceive this as a larger cache and the ability to use its contents in ways which contribute towards the system’s tasks. In practice, it also extends the immediate context available to neural networks and thus their ability to relate their inputs to it to limit their ambiguity.
As I understand it, this enables, for example, for text generators to ‘remember’ larger chunks of what they have already written and to account for each word in that context to calculate the most probable next one. They can also account for their own earlier outputs when an user requests one to, for example, clarify what it meant.
Accounting for a wider context like this reduces issues related to internal context blindness which has in part made earlier attempts seem so artificial. As a result, such outputs do not, for example, usually suddenly change the subject only because some part selected for the output is now more compatible with samples related to a different topic. AIs with a chat UI do not also as easily ignore what they have already expressed and, for example, repeat themselves when asked to correct their answers.
Attention like this cannot extend to context outside the medium, however. This distinction between an internal context and an external one is important because even an unlimited attention for internal context does not yield any ability to account for external context. That would require a distinct technology.
Applications of AI for documentation
In my opinion, the primary application for generative AI in technical documentation is not having it generate text. In order to train one to sufficiently match the style of a company’s manuals, you would need to use a training sample with enough documentation already in that style.
Instead, I see the following uses as more helpful:
- Word suggestions,
- Simplifying filters,
- Transitions in format, and
- Search bots.
This list is not exhaustive. You can also use AI to, foe example, help with translations or to write style sheets. These other applications highlight the limitations of AI, though, which ultimately requires that someone with the kind of expertise that the AI was meant to supplement reviews its results. Additionally, such reviews for these tasks can be as laborious as doing the work yourself if not more so.
Word suggestions
One limitation of the methods used to reuse content in structured authoring is how you generally cannot control the uniformity of the formulations inside corresponding but not identical elements. You can thus use them to create elements that you can embed in layers but these elements cannot be adjusted internally outside the use of other, smaller elements. Language is more flexible than this. As such, fully uniform elements like this cannot adapt to natural grammatical variation between languages when it comes to controlling word choices inside individual sentences.
In principle, an AI trained with your prior documentation could provide suggestions to finish such schematic portions based on the correct templates after a few words into them. It could also technically go and edit such portions based on new instructions in a way which would work better than finding and replacing all matching parts.
I gather that such services already exist as plugins. For example, the browser plugin for Grammarly is compatible with the text editors used in DoX CMS. I have not thus far tested if such plugins can be trained to make their suggestions based on the style of your existing technical documentation.
Simplifying filters
In addition to text, documentation often contains images and even videos. For the best results in terms of communicating, information security, and participants’ privacy, such parts should be simplified to only show the situationally relevant parts. Other information should thus be blurred or covered for it not to distract users or to, for example, cause unnecessary risks to information security by showing unedited backgrounds.
Generative AIs are able to edit images and videos according to your instructions when those are fed to them. Such processing done by a human can take considerable amounts of time even when the changes involved might be relatively simple, such as obscuring other parts of UI or values inside a system that a screenshot shows. This is often not done because it provides additional benefits insted of being a requirement for content deliveries to users. AI could thus bring additional value by enabling such further processing for a relatively minor time investment.
Transitions in format
AI can both read text aloud and add subtitles to speech. Their ability to generate videos based on descriptions seems to also develop rapidly. With such help, those responsible for documentation can thus transition documents in one format to other formats with little effort.
When it comes to reading documents in a structured format, I am somewhat worried over an AI’s ability to account for structured sections such as lists and tables appropriately. A minimum requirement for this would be that the documents were originally written in a format suitable for being read by an AI. A prior example of such a challenge is how screen readers process documents with invisible tables as scaffolding for their layouts.
Search bots
A proper search function is crucial for the usability of any even slightly longer documents. It ensures that users find the information that they need with minimal effort.
Even though AIs so not comprehend causality, for example, they remain considerably more lenient in terms of searches than options like strict (key)word searches. As such, users need not necessarily know in advance the search terms which they must use to locate the correct content. They can also formulate their queries more naturally as questions instead of such questions needing to first be translated into search terms.
An AI can also base its answers on separately documented sections. At times, the situationally required information may be found in several parts of a document or even between documents. An AI can find the situationally useful details from each such section instead of the user needing to locate each of them separately based on what further information they can infer is needed to complement what they have already found.
For further information on how to improve your documentation’s readability from an AI’s perspective, see here.
Restrictions of AI
What the providers of AI technology promise are only one side of this coin. Because it might as well be magic to most users and because this technology is capable of rapid advancements based on the size of available training sets and the autonomous learning that adversarial networks, for example, enable, many of its proponents are willing to paint bombastic pictures of the future in the air. The technology’s capacity to realize what is promised is not given, however, despite its rapid advancements. The fundamental technologies which have enabled those advancements remain the same, and they have limits of their own.
From the standpoint of documentation, the most relevant such limitations are how the processing done by such AIs involves meaninglessness, detachment, and dependence. Each such deep limitation manifests in various ways.
Meaninglessness
The source of the meaning of signifiers like words remains too grand a philosophical question to be addressed here, and it lacks agreed upon answers regardless. The processing done by an AI does not fulfill the minimum requirements of pretty much any perspective on the content having meaning to those systems. Most schools of philosophy would not even consider an AI to be in possession of a perspective.
That the processes content is without meaning to the one processing it, which refers to an AI here, is a huge restriction, though. It is partially responsible for the so-called hallucinating that AIs do. However, I will address that problem separately in relation to the issue of detachment. Meaninglessness, specifically, gets expressed in two immediately recognizable ways, which display the limitations of the principles behind AI.
An AI never asks clarifying questions.
An AI does not correct its answers but answers anew, instead.
Why does an AI never ask clarifying questions? This would be an obvious method to reduce the number of incorrect answers, when a prompt involves some form of uncertainty or vagueness. However, an AI does not process a prompt’s content. They only process a prompt’s form. They simply cannot recognize a need for clarification. If you manage to make one ask further questions, it is a matter of a statistically believable response to a prompt like that being a question like that. This is not a question proper but an answer in the form of a question because an AI does not ask to receive answers. It has no interest in users’ answers because that is beyond it.
In principle, you could add a separate system which optimizes prompts before they get processed with the help of the primary technologies. Such a system would ensure that the prompts actually allowed to be processed by the primary processing system would sufficiently account for its requirements. Such a system cannot use the same technology, though, because it would specifically be needed because these systems only process prompts as strings of symbols. Since these symbols have no significance to them, they cannot identify abnormalities or deficiencies at the level of content. For example, an absurd question such as ‘What did president Abraham Lincoln think of the Internet?’ is no different from a genuine question to an AI, and it will thus try its best to answer it. It is possible that it could respond by saying that Abraham Lincoln lived before the Internet was invented. However, doing so could be hindered by memes, for example. A widespread meme jokes that Abraham Lincoln once said that you should not trust everything that you read on the Internet just because it is presented as a quote from someone respectable. An AI does not recognize the context for such memes, which presupposes it to be common knowledged that Abraham Lincoln did not know what the Internet is.
This leads to the other restriction that I mentioned above: AI does not correct its answers. You can request them to correct their replies, and they will apologize and form a new response which excludes alternatives that approximate the original answer. Alternatively, they can insist that they are correct because this would be a common and thus a believable response to being doubted. Regardless, this is not a matter of it correcting its anwer, because the goal is not increased accuracy but rather, acceptability to the recipient. A user who disagrees could tell an AI that the correct answer is incorrect and have it ‘correct’ its response. Imagine that instead of an AI, you would have a person who ‘corrects’ their answers by simply responding in a different way but without checking any sources related to the subject and without making an attempt to find a direction for their reply besides your direct feedback. If it was not simply a matter of indifference towards the truth in that situation, such behavior would seem clearly pathological.
Some of these behavioral patterns in AI that I just described have likely been ‘fixed’ in ways not unlike the fixes to well-known uncanny features of the images that they generate. As I understand it, such corrections consist of a combination of hard-coded filters and retraining with the help of discriminators which were adjusted to be more discerning in those respects. Regardless, such further development does not change the root causes of these issues, which makes them remain a real risk. The most basic and the most advanced AI only differ in these respects in their the degree to which they are able to satisfy their users. They all remain the ‘stochastic parrots’ described by professor Emily M. Bender and her colleagues. They merely blindly mimic human behavior based on what activity grants them crackers. You can read more on Bender’s claims here, for example.
Detachment
The detachment of AIs refers to their inability to account for any kind of context outside the content itself that I mentioned above. They cannot relate the forms of media that they process to details external to those media. The processing that they do is only a matter of shuffling and chopping the material in these formats in their training samples and then recreating approximations of its contents.
Discussions about the tendency to hallucinate that AIs have involve this inability to process information external to available forms of inputs. Distinctions between truth and falsity are possible to us as humans because we can compare statements with facts beyond them. Of course, people’s ability do do so also deteriorates when dealing with differences that can only be identified through special expertise or which otherwise do not present themselves as part of a person’s direct experience, for example. However, AI provides an extreme example of this. For them, nothing past provided inputs exists. As such, they cannot connect parts of that input to what they would correspond to in reality to identify their referents.
In this respect, they are not hallucinating but rather, being poor performers. You must remember that generative AI was trained to try its best to deceive discriminators which compare their creations to samples available to themselves. AI discriminators are more discerning than people in terms of details which are more evident to an AI’s operating logic, and they allow for significantly faster training with more cycles. People are also discriminators, though, and at least at times, we pay attention to details such as whether something fits the facts. In a way, this resembles an admittedly masterful actor or an infiltrator who has only studied for their role by reading text in a language unknown to them. They do not understand what they say, and they lack all cultural context for those expressions. When they make errors, it is not a matter of hallucinating. They simply stumble and fumble blindly while trying to remain convincing. Like Matt Pearce states in this podcast interview with Adam Conover, AI provides an excellent illustration of Plato’s allegory of the cave with its prisoners who may only interpret reality through shadows cast on the cave wall and whose whole sense of reality consists of those shadows.
Detachment also entails that AIs cannot incorporate influences beyond the forms of available inputs to express their creativity. At times, the kinds of recombining of prior samples that they do are defended by claiming that this is no different from human creativity. Such creativity is also largely about combining influences, some of which are others’ creations. However, an important distinction is how people can combine influences freely from distinct sources: across different media and random experiences. Such influences may also manifest in ways that change and adapt them past simply changing their exact formulations. For example, in the case of a person, Plato’s allegory of the cave which I mentioned above could inspire a story where humanity is trapped in a virtual illusion which fails to correspond with reality. The perspectives of those inside the story would treat such disparities as normal until the connection between them and the reality beyond the illusion is revealed to them. An AI could perhaps also construct an outline which references the allegory of the cave and utilize existing creations which involve that connection to express it. What it could not do, however, would be a novel application of this reference because its fundamental goal is to generate outputs which resemble what is familiar. Such creativity which is fundamentally about the creation of novel interpretations, instead, is normal to people but diametrically opposed to how an AI operates.
Of course, at this point, some of us will remember images generated by AI, the likes of which we had never seen before. I do not mean their uncanny failures but rather, creations treated as successes which involve, for example, familiar subjects worked into fractal patterns or blending of subjects in ways which none has ever done before. These creations being unique has available explanations, though. The most important factor in explaining them is just that AI applies an unfamiliar procedure to generate novelty. The alien nature of this procedure becomes more obvious the less obvious the output that corresponds to a prompt would be. This in turn leads us to the other factor which explains this: the creativity of the prompt writer. Once you account for their limitations, AI can be applied as a formidable creative tool. This admission may seem to contradict my earlier claims about the creative limitations of AI but the AI itself need not be any more creative than that in this situation. It is an algorithm and thus, its output is determined by the provided inputs. In practice, you need to know how to translate creative ideas with its requirements in mind, and this is not achieved with general descriptions such as ‘the Matrix but with a virtual world more directly based on Plato’s allegory of the cave’. This distinction is important to observe in particular when some expect AI to be able to replace creative workers. Of course, creative workers have always done more than just creating content which makes that expectation unrealistic from the start.
Dependence
Dependence relates to how the limits of an AI are determined by the samples used to train it. This is no different from other learners, of course. In the case of an AI, it will be fed vast amounts of material before a sufficient level of performance is achieved. For example, the development of current AI tech was dependent on large language models and image banks made with content taken online. If this level of performance is dependent on such content shared by others, it is also vulnerable to changes in such content.
An ironic challenge in this respect is how these systems becoming more widespread may in itself make their long term performance worse. The greater the proportion of AI-generated content in their input becomes, the more new AI-generated content will resemble their former outputs. In the long term, it is possible that such incest will give our new AI overlords the Habsburg jaw or hemophilia. A fix may well be found through such parallels to genetics and related methods to control the variety of inputs. Then again, the project has been eumemetic from the start as there is proof that the training samples were selectively created from the works of named artists and high quality publications. It is quite possible that the inclusion of amateurs’ works in training samples would manifest in equally harmful-seeming tendencies as replicating the earlier creations of AI. It is also a fascinating thought that perhaps the inputs allowed to be used to train AI must be filtered with programs that identify AI-generated content. Such discriminators would ultimately do the same thing as the discriminators used to train AI, though. In other words, these AI were originally trained to deceive such discriminators and as such, the filtering discriminators would need to always remain ahead those used for training or to apply a different method of identification from those ones.
Another issue with this need to use as large inputs as possible is how it exposes these AIs to being sabotaged. Because AI often makes use of content without permission, visual artists in particular have resorted to various content poisoning tools while questions about copyright law remain pending. Such tools such as Nightshade add features which are indiscernible to humans but mislead AI to content. In the case of writing, examples include hidden parts which directly instruct any AI on how to respond to questions about this content. These issues would not be an issue with AIs which were trained with only controlled contents such as a company’s own documentation. As I understand it, most such solutions are still ultimately based on AIs which were trained with public content, though, as this ensures a sufficient baseline for them. As such, they can also be poisoned like this if AI developers continue to tap training content without consent.
Of course, this craving for content used to feed such dependence is also a problem when the tool in question also saves your prompts and other training data that you provide. This poses a huge risk for information security. All documentation is most certainly not meant to be public, and targeted prompts allow you to extract direct references to their training data from these systems. This is a key factor in the New York Times legal case against OpenAI and Microsoft, as you can read here. Likewise, Samsung provides an example of an information breach caused by the use of AI, as their workers apparently fed source core to ChatGPT, as you can read here. When an error like this occurs, it is impossible to undo.
Summary
Even from the perspective of an expert in AI, they currently remain mostly a means to automate specific parts of the writing pricess such as finding sources, generating first drafts, and editing it for effect. Human writers remain necessary for further processing of such content because AI is simply too unreliable in terms of meaning in particular. They can be used to support writers in getting started, though, or in whichever ways those writers need such support. In part, this is a matter of removing obstacles other than those which are inevitable for such work as a means of increasing its accessibility.
The operating principle of these systems is that the neural networks that they use are trained semi-autonomously with the help of big data, the content of which an adversarial network with a generator and a discriminator try to either mimic or discern, respectively. This is not the only method to achieve such results, but each such method allows for the seeming creativity of related AI within the bounds of gaps in available samples as approximations of content that matches the samples and their parts. Modern AI is also more believable to humans because of a higher attention span which allows them to account for continuity within larger wholes. This involves both greater internal consistency and continuity between consecutive prompts.
As far as I am concerned, the best ways to use AI to support documentation are word suggestions, simplifying filters, transitions between formats, and search bots. All of these are forms of labor where AI provides especially suitable tools and where the weaknesses of AI such as the inability to discern facts are mostly non-considerations. Word suggestions which have been trained with your content let you to uniformly repeat inline sections. Simplifying filters let you cover parts of images or videos which would be distracting or otherwise unnecessary but able to accidentally cause issues with privacy or information security. Both these filters and transitions between formats improve delivery quality but they require further labor which is not necessary for deliveries and would thus be hard to justify if it was to be done only by those responsible for documentation. For example, expressing the same content both as text and as a video would often involve almost doubling the workload, and it would also require special skills related to each form of documentation. Searchbots, on the other hand, let the end users search for information in a more natural way because their queries are not limited to matching strings. In principle, they can also combine information from otherwise separate parts in response to situational needs.
AI has its limitations, and some of those relate to how the technology operates rather than just the implementations thus far. As such, related issues cannot be overcome with further development of the same technologies. That would require novel solutions, instead. Such challenges include how content is meaningless to AIs, AIs being detached from reality, and their dependence on the provided inputs. Since an AI can only process inputs’ forms rather than their meaning and referents, it cannot redirect its answers beyond excluding already attempted answers based on immediate feedback. They do not ask for clarification to help direct their attempts, because the only thing which they can do is to generate convincing responses to prompts. In part, this is a matter of them being detached from reality which prevents them from using external context past the contents themselves. An AI’s responses are indifferent to the reality of the matter because it cannot verify any claim. They are all imitators which play their roles to the best of their ability based on nothing but having heard a language which is completely foreign to them. As such, they also cannot adapt influences from beyond already familiar content in novel ways unless their users do this for them inside their prompts. Such dependence on available samples made by humans also manifests in other ways. The proliferation of AI-made content can thus actually lead to incestuousness which weakens their outputs going forward. Efforts to also use as much data as possible to train them also increase the risk of inputting content which is poisoned against AIs. That dependence also entails that they crave for input as samples from their users which can lead to risks to information security.