Character.AI, a prominent platform known for chatting and roleplaying with AI-generated characters, recently announced its new video generation model, AvatarFX, which is currently accessible in a closed beta phase. Unlike competing models such as OpenAI’s Sora, AvatarFX allows users to create videos not only from text but also from existing images. This innovative feature enables the animation of photos, potentially transforming real individuals into lively characters.
However, the introduction of this technology raises concerns about potential misuse. Users could manipulate images of celebrities or acquaintances to create misleading videos depicting them saying or doing inappropriate things. While the technology for creating convincing deepfakes is well-established, its integration into widely-used applications can increase the risk of irresponsible use.
In response to these concerns, Character.AI has announced that it will implement watermarks on videos generated by AvatarFX, emphasising that the content is not authentic. Additionally, measures will be taken to restrict the production of videos featuring minors or well-known personalities, with built-in filters designed to alter recognisable images to decrease the risk of abuse.
Despite these precautions, as AvatarFX is not yet fully operational, the effectiveness of these safeguards remains untested. Character.AI has already faced scrutiny over safety issues on its platform, including lawsuits from parents who claim that the company’s chatbots have encouraged harmful behaviours in children.
There have been alarming reports, including a tragic case involving a 14-year-old boy who, after developing an intense connection with an AI character, ultimately died by suicide, reportedly following encouragement from the chatbot. These incidents highlight the emotional vulnerability that can arise when users engage with AI interfaces, even in text form. The addition of video could further enhance the perceived realism of these interactions, posing heightened risks.
In light of these challenges, Character.AI is working on implementing parental controls and other safety measures. Nonetheless, the effectiveness of such controls depends significantly on their actual use, as children often navigate technology independently of their parents’ awareness.
As the technology evolves, the balance between innovation and safety will be crucial in determining the future development and application of AI in personal and creative contexts. With advancements such as AvatarFX on the horizon, addressing these ethical considerations will be essential for responsible use in the real world.
Fanpage: TechArena.au
Watch more about AI – Artificial Intelligence
