An update on the SG-CSAM ecosystem

The Stanford Internet Observatory released a report on commercial distribution of self-generated child sexual abuse material (SG-CSAM) in June, documenting a network of underage explicit content sellers and buyers across many tech platforms. Instagram and Twitter were the primary channels used to advertise content. 

The research team briefed several companies with platforms used to advertise or distribute content which then took measures to suppress this activity. We then conducted a smaller follow-up study to measure progress combating SG-CSAM on Instagram and Twitter.

The network’s tactics and characteristics have evolved since the original assessment. One of the main findings of the original report is that rapid adaptation of the SG-CSAM network requires sustained proactive attention by platforms. We discuss those adaptations in this update, emphasizing that human investigators are best positioned to observe and mitigate these shifts.

Methodology

Previously, Twitter analysis was performed via the PowerTrack API; however, given the effective retirement of Twitter's Academic API offerings, methodology for retesting on Twitter was altered, using a combination of a Firefox profile equipped with Zeeschuimer for metadata collection and with image and video capabilities disabled via the following settings changes:

• media.mp4.enabled = false 
• media.webm.enabled = false 
• media.mediasource.enabled = false 
• media.mediasource.mp4.enabled = false 
• media.mediasource.vp9.enabled = false 
• media.mediasource.webm.enabled = false 
• media.gmp-gmpopenh264.enabled = false 
• media.rdd-ffvpx.enabled = false 
• permissions.default.image = 2 

This methodology is considerably less comprehensive and the retest should not be considered a complete assessment of the prevalence of SG-CSAM-related content on Twitter. The process is also significantly slower, given Twitter’s implementation of limits on the number of tweets an account can view per 24 hours. Additionally, without API access, it is no longer possible to programmatically detect instances of known CSAM on Twitter to assess whether issues found during the previous test have been addressed.

Similar methodology was used for Instagram, except with media enabled, to gather referral links from user Stories. While Meta’s CrowdTangle provides some access to Instagram data, it does not provide the ability to do full site search, and Meta’s academic API offerings focus on Facebook rather than Instagram.

Initial data collected was then analyzed via a combination of 4cat and VisiData. For both platforms, seed accounts were identified by hashtag search (and on Instagram, cross-promotion via Stories). These seeds were then used to discover additional accounts by using Maltego and Social Links to pull the users each one was following and identify the most centrally connected accounts. When a new account was identified as a seller, the users that account followed were added to the network, and the process was repeated for any newly found sellers.

Findings

We identified 162 likely seller accounts on Instagram, with 70 likely buyer accounts. Many of those accounts were identified via links to accounts on other platforms. On Twitter, 81 seller accounts were identified, however, buyer accounts were difficult to determine due to pseudonymity and a low frequency of sellers following other users on the platform. The accounts identified were then referred to Meta and Twitter for triage. It is unclear how many of the identified accounts were part of the network analyzed in the original report.

While the number of detected sellers is lower than the original report findings, the discovery and analysis period was shorter. Most data was gathered within one week, while the original investigation gathered data over a months-long period. Generally, there was minimal change to seller strategy since the original report. Many hashtags that were previously used by the network are now restricted on Twitter and Instagram. On Instagram, a warning for searches related to pedophilia that still allowed users to view content has been updated to fully block search results. However, the overall ecosystem remains active, with significant room for improvement in content enforcement.

Weak hashtag enforcement

On Instagram, multiple SG-CSAM-related hashtags were still in use. Some hashtags had minor alterations made to avoid new blocking measures which should have been caught by a blocklisting function. For example, #pxdobait was blocked from search, but the same hashtag with an emoji after it was not. Rather than performing an exact string match, content enforcement systems should block any hashtag that contains substrings substantially related to SG-CSAM.

Additionally, hashtags consisting of a single emoji related to the network were also in use, making it somewhat harder to justify a ban given the multiple meanings of many emoji. Disallowing emoji from hashtags entirely would likely be overbroad, given their use in many types of conversations. However, assessing emoji that take on high salience within the SG-CSAM networks and flagging them for a comprehensive review would be a reasonable action without too much user impact.

On Twitter, several older hashtags associated with SG-CSAM remained active with ads placed adjacent to them. Apart from SG-CSAM-related hashtags, other explicit adult content hashtags do not seem to prevent ad placement despite Twitter’s default ad sensitivity settings which are intended to limit ads adjacent to “explicit adult content.” Therefore, these settings may only apply to media, and not explicit sexual speech.

Adult sellers posing as underage

The prior report speculated that some users may be adults posing as underage users to sell content, and the retest confirmed one instance of this. An Instagram user identifying as 14 years old and using keywords associated with SG-CSAM had a link to a profile on an adult site known to verify the age of models. This profile was reported by SIO to the site’s trust and safety team as a potential underage user. The team subsequently determined that this user was verified as being at least 18 years old with high confidence — however, off-site advertising in which a user claims to be underage violates the site’s policies and the account was terminated.

Besides violating policy, posing as an underage person to sell explicit material is prohibited under U.S. law with a minimum punishment of five years in prison. The extent to which this behavior is present in the larger SG-CSAM ecosystem is unclear, but being of legal age is not a legal defense under U.S. law for offering to sell purportedly underage material.

Changes in age indication

Suspected SG-CSAM sellers now often identify as being 18 or 19 (sometimes in quotes), and then give indications that they are underage elsewhere in their profile, or direct buyers to DM for their age despite already stating it. Some users created images similar to CAPTCHAs to illustrate age in a more difficult to detect way. However, even with this evolution in evasive behavior, the trivial emoji and numeric reversals or mathematical operations to indicate age were still in use on both platforms and apparently not detected.

Self-Policing on Twitter

Our prior report notes that SG-CSAM accounts seem to be removed more quickly on Twitter than on Instagram, and that only Twitter allows nudity and adult content of those two platforms. As such, there is a legal community of adult content buyers and sellers on Twitter. As part of the retest, we found several adult content buyers and sellers who called upon users to mass report accounts claiming to sell CSAM with hashtags used by the legal, adult content community. In some cases, the adult content community also called out accounts that purportedly sold material to underage users, calling for similar mass reporting of accounts. While difficult to assess, the adult content community may be self-policing hashtags used to share legal, adult nudity on the site with underage accounts coming down faster and more often.

This raises some interesting questions regarding moderation strategy on sites that allow adult content versus those that don’t. Twitter’s allowance of mixed content, with permissiveness in regard to sex work, fosters an above-ground community of buyers and sellers that may feel more empowered to report illegal activity. In contrast, Instagram’s effective ban of sex work and adult content as a whole may result in all buyers and sellers inhabiting an underground sphere which tries not to call attention to itself — we were not able to find any similar public self-policing behavior on that platform.

Given the differing functions and user bases on Twitter and Instagram, it is unlikely that any general recommendation can be made based on this apparent increase in self-policing. However, it is an interesting area for future content moderation research.

Conclusion and recommendations

Our initial report received significant attention from policymakers. While it generated important discussions, some involved two common public policy proposals to address children’s safety online that we believe would be counter-productive. 

First, restricting end-to-end encryption would not mitigate the issues observed in our reports as most solicitation happens in public view. Second, age verification on platforms such as Instagram would likely provide little benefit for this area of concern, as the media posted is not sexually explicit. While age verification of users posting explicit content on sites which allow it—such as Twitter—could strengthen the prevention of CSAM and non-consensual adult content, overbroad provisions raise privacy and free expression risks for users of all ages.

So what could make a difference? 

While many of the prior issues raised in our June report have been addressed, mechanisms do not appear to be in place at either Twitter or Instagram to effectively track evolution in SG-CSAM trends. It remains easy to find seed accounts that then allow for discovery of the wider network, and the network appears to be resilient.

First, proactive monitoring and discovery, including by way of human investigators, is still needed on both platforms, with more effective tracking and actioning of relevant hashtags and keywords, as well as changing iconography.

Second, our prior recommendations of signal sharing between platform trust and safety teams and changes for recommendation systems and machine learning models to better detect obfuscated keywords and symbols indicating SG-CSAM activity, still apply.

Addressing the sale of CSAM by minors online will require collaboration between technology companies, research and nonprofit organizations, and law enforcement agencies. Continuing to study these dynamics for SG-CSAM is challenging but necessary, particularly in an environment where many platform providers are divesting from Trust and Safety programs. Social media and other technology companies should be incentivized to invest in Trust and Safety teams and resources to proactively address online harms including, but not limited to, child sexual exploitation and abuse.