Survivor-First Technology: A Guide for Evaluating AI Vendors

Questions to ask any technology vendor you work with — including us.

How to use this guide

Technology built for survivors is being adopted faster than the standards to evaluate it. Well-intentioned tools can still cause harm — through what they collect, what they report, who they answer to, and what happens when they get it wrong.

This guide is built to be used repeatedly. Bring it to a demo. Send it ahead of a procurement conversation. Work through it with your team before you sign anything.

Each section gives you the question to ask, why it matters, and what a strong answer sounds like versus one that should give you pause.

One principle underneath all of it: survivor-first means the person who uses the tool ends the interaction with more information, more choice, and less exposure than they started with. Every question below is a way of testing whether that's true.

A note on answers: a vendor saying "I don't know, let me find out" is not a red flag. A vendor who is vague, changes the subject, or answers a different question than the one you asked — that is worth paying attention to.

When you hear a technical term

You do not need to be technical to evaluate these tools. You do need to know when an impressive-sounding answer is answering a different question than the one you asked. A few of the most common:

"It's encrypted" / "end-to-end encrypted." Encryption protects information from being intercepted or read by outsiders (e.g. hackers). It does not stop the company itself from accessing it, does not stop it from being handed over under a court order, and does not mean it isn't being kept, used, or sold by the company.

"We're SOC 2 compliant." This is a security audit — it confirms the company follows its own stated procedures for protecting data. It says nothing about what they collect, how long they keep it, who they share it with, or whether the product is safe for survivors. A tool can be fully SOC2 compliant and still not be survivor-centered.

"We're HIPAA compliant" or "HIPAA-aligned." HIPAA governs medical records held by healthcare providers and their partners. Most survivor-services technology tools are not required to be HIPAA compliant, so "aligned" often means "we chose to follow some similar practices." Ask which practices, specifically.

"It's anonymous." Ask what that means in practice. Some tools collect no identifying information at all. Others collect it and then remove names — which can often be reversed. Follow up with: what exactly is collected, and could a person be re-identified from what remains?

"The AI is trained on it" / "it learns from usage." This means what people type is being fed back into improving the model. Ask directly whether conversations are used for training, and if the answer is yes, what that means for information a survivor shared.

"We do adversarial testing" / "red teaming." This means someone deliberately tried to break the tool to test it — tricking it into saying harmful things, bypassing its rules, or accessing data it shouldn't. It's a good sign when it's real. Ask who did it, when, what they found, and what changed as a result.

Before you evaluate anyone's product: know your own obligations

If your organization receives VAWA, FVPSA, or VOCA funding, you are already bound by federal confidentiality requirements — and those obligations apply to the technology you choose, not just to your staff.

The confidentiality of survivor information collected by government-funded victim service providers is governed by federal law (VAWA 34 USC §12291(b)(2), FVPSA 42 USC §10402, VOCA 28 CFR 94.115) and by the laws of most states. These provisions apply to all grantees and subgrantees, and they broadly bar disclosing information about a survivor receiving services.

Three points that bear directly on vendor selection:

• Consent cannot be a condition of service. Grantees are prohibited from requiring survivors to sign a consent to release personal information in order to receive services. If a product only functions when someone agrees to broad data sharing, it may put your program out of compliance — not just out of alignment with your values.

• Survivors control their own consent, including minors in many cases. VAWA 2013 strengthened survivor control over when to consent to sharing, and clarified that minors permitted to receive services on their own may sign their own consent. A tool that routes disclosures around the survivor's decision is working against the statute, not just against good practice.

• Privilege is a separate protection from confidentiality. Where an advocate holds privilege, a court generally cannot compel disclosure of what a survivor shared with them. Ask whether a vendor's data practices could create records that sit outside that protection — information held by a technology company may not carry the same shield that information held by an advocate does.

A caution: these are general descriptions, not legal advice, and application varies by funding stream, state, and circumstance. Confirm your own obligations with your own counsel — and be skeptical of any vendor who tells you definitively what your compliance obligations are.

A resource worth using: NNEDV's Safety Net Project maintains a Confidentiality Toolkit built specifically to help providers understand obligations under VAWA, FVPSA, VOCA and related privacy laws, along with a dedicated guide on selecting a database vendor covering purpose, confidentiality, data security, and program capacity. It is the most thorough peer-authored resource in this space and pairs well with the questions below. (techsafety.org)

Evaluation Questions:

1. What safeguards protect the person using it?

Why this matters: For someone whose device is monitored, or who is being controlled by another person, using a tool at all carries risk. The safeguards aren't a feature list — they're the reason it's safe to reach out in the first place.

Ask specifically:

• If someone's device is monitored, does using this leave a trail that could be found?

• Can someone close it instantly, and does that actually clear the history behind it?

• What happens if the tool says the wrong thing to someone in crisis — and how would you know it happened?

• Does it try to keep people engaged, or does it point them toward real human support?

Strong answers sound like:

• Specific mechanisms, not adjectives. A quick-exit function, guidance on private browsing, automatic session expiry.

• An honest acknowledgment that the tool can get things wrong, paired with a described process for catching and correcting it.

• Language about connecting people to humans and services, not about keeping them in the product.

Answers that should give you pause:

• "It's completely safe" with no mechanism described.

• No answer for what happens after a bad response — or a claim that bad responses don't happen.

• Engagement metrics offered as evidence of success (time in app, return visits, session length). For a crisis tool, high engagement may be a warning sign, not a win.

2. How is their information protected from being used against them later?

Why this matters: This is the question most often skipped, and it is the one with the longest tail. Information collected today can surface in a custody case, a criminal proceeding, or an abuser's hands years from now. "We keep it secure" answers a different question than "could this be produced later."

Ask specifically:

• What is collected — including anything that could reveal someone's location?

• How long is it retained, and who can see it during that time?

• Can it be permanently deleted, and can the person themselves request that?

• Could it be subpoenaed, handed to law enforcement, or pulled into a legal proceeding?

• Is any of it shared with a third party? If so, exactly what is shared, with whom, why — and can you opt out?

Strong answers sound like:

• A specific retention period, and what happens at the end of it.

• A clear answer on what identifying information is not collected at all — the strongest protection against disclosure is data that doesn't exist.

• A direct answer to the subpoena question, including its limits. A vendor who has thought about this will not be surprised you asked.

• Named third parties and a specific reason for each.

How to judge a retention answer: there is no universal standard, but the direction matters more than the number. Shorter is safer. Retention measured in days is easy to justify — it usually means the data exists only long enough for quality review, then goes. Retention measured in months should come with a clear reason. Retention measured in years, or described as indefinite, means the vendor is keeping information about survivors long after any operational need, and everything kept is something that can eventually be requested, breached, or disclosed. Ask what specific purpose the length serves. If the answer is "in case we need it," that is not a purpose.

Answers that should give you pause:

• "We're fully encrypted" as a complete answer. Encryption protects data in transit and at rest; it does not prevent lawful disclosure.

• Vagueness about retention — "as long as necessary," "per our policy" — without a number.

• Discomfort or deflection on the subpoena question.

• Third-party sharing described only in general terms, or buried in a privacy policy you're pointed toward rather than told about.

Worth asking alongside this: where your advocates hold privilege, a court generally cannot compel disclosure of what a survivor shared with them. Records held by a technology vendor may not carry that same protection. Ask whether using this tool creates a copy of survivor information that sits outside the shield your own staff would have.

3. How does it preserve the person's control — including whether anything is ever reported without them?

Why this matters: A tool that reports on someone's behalf takes away the one thing many survivors have left, which is deciding when and whether to disclose. Research consistently shows that automatic reporting without consent often backfires, and people disengage rather than come forward. Beyond the harm to the individual, a report generated from technology use data without a person's participation is frequently unusable to the people receiving it.

Ask specifically:

• Does it report anything automatically without the person choosing? Can it be turned off?

• Does it push toward a particular decision or support the person in making their own?

• Does it diagnose people or apply clinical labels, or does it leave that to a qualified professional?

• Does using it create any reporting obligation for our staff, or interact with our existing mandates?

Strong answers sound like:

• A clear statement that nothing is reported without the person's decision — and an explanation of why that's the design, not just that it is.

• The tool informing and offering options rather than directing a specific outcome.

• A refusal to diagnose, with a stated handoff to qualified humans.

• On mandatory reporting: a clear description of what the tool does and does not do, and a direct recommendation that you confirm your own obligations with your own counsel. A vendor who tells you definitively what your staff's legal obligations are is overstepping — those vary by state, by role, and by what is disclosed.

Answers that should give you pause:

• Automatic reporting described as a safety feature, without acknowledgment of the tradeoff.

• No option to disable automated escalation.

• The tool assigning labels or assessments that belong to trained professionals.

• Confident legal advice about your compliance obligations from a company that is not your lawyer.

4. What is the AI doing, how was it built, and what are its limits?

Why this matters: "We use AI" covers an enormous range — from a system built with survivors and advocates over years, to a general-purpose chatbot with instructions attached. Those are not the same product, and they do not fail the same way.

Ask specifically:

• Does it use AI at all? If so, for what part of the experience?

• How was it built — who informed the design, and what expertise shaped it?

• Is any of the data people enter used to train the model?

• Can it be talked into ignoring its own rules — through a hypothetical, a roleplay, a "just pretend"? Has anyone tried on purpose?

• How does the AI handle sycophancy (the tendancy for AI to agree with users)?

• Does it stay within its purpose, or will it answer anything?

• Does it tell people it's AI, and that it can be wrong?

Strong answers sound like:

• A specific account of who shaped it — survivors, advocates, clinicians — and how.

• A direct "no" on training using chat data, with an explanation of how the product improves instead.

• Evidence that someone has actively tried to break it, and what changed as a result.

• Clear scope limits, and a description of what it refuses to do regardless of how it's asked.

• Disclosure to users that they are talking to AI, that it can make mistakes, and an easy way to report one.

Answers that should give you pause:

• "It's AI-powered" with no detail about what that means in practice.

• Conversation data used for training, framed as making the product better for everyone.

• No adversarial testing, or testing described only as "we tested it."

• No acknowledgement or strategies for mitigating hallucinations and sycophancy.

• A tool that will answer questions far outside its stated purpose — scope drift is often how a crisis tool becomes something else.

• No upfront disclosure that it is AI.

5. Who stands behind it, and how do we reach them?

Why this matters: You are not only choosing software. You are choosing an organization you will need to reach when something goes wrong, and whose priorities will shape the product over time.

Ask specifically:

• What lived experience experts and organizations already trust the tool?

• Who reviews conversations or flagged issues, and are they trained in this work?

• Has anyone independent tested it — for security, and for how it responds?

• What languages does it work in?

• If something goes wrong, who do we call, and how quickly?

• What is the organization's model — who funds it, and what are they accountable to?

Strong answers sound like:

• Credible organizations and leaders who endorse the tool.

• Named roles and real qualifications for whoever reviews content.

• Independent testing by an outside party, with findings that led to changes.

• A specific escalation path and a realistic response commitment.

• Transparency about funding and incentives.

Answers that should give you pause:

• Review done entirely by automated systems, or by staff with no training in this field.

• No independent testing.

• No clear escalation contact.

• An unwillingness to discuss the business model or who the organization answers to.

Before you sign

• Did the vendor answer the question you asked, or a different one?

• Did anything get described as impossible, when it's actually just unlikely?

• Would you be comfortable explaining this tool's data practices to the people who will use it?

• If this goes wrong in a year, will you have been able to say you asked?

Questions about any of this — including how our own tools measure up — reach us at our contact us form