6 Questions to Ask Before You Build or Buy an AI Tool for Skilled Nursing

Picture your company's new headquarters, designed from the ground up by AI. The floor plans are complete. The renderings are flawless. No structural engineer ever signed off on them.

Would you take the corner office on the top floor?

That isn't a hypothetical anymore. Zevi Britz, one of the engineers who builds AI on healthcare data at Megadata, recently heard from his brother-in-law, an architect, that the latest AI models can now draft complete plans for an office building. They look beautiful. And yet:

"No CEO is taking the office on the top floor of something that was built like that, even though it looks beautiful on everything."
Zevi Britz, Senior Software Engineer, Megadata

You would never move your team into that building. But skilled nursing leaders are being asked to do something very similar with software every week.

Admissions tools. Referral tools. Staffing tools. Rehospitalization tools. Many were built in a matter of weeks, and on demo day they look just as polished as tools built over years. The difference shows up later: when your EHR vendor changes a field, when you acquire three more buildings, when the census number in one report doesn't match the one in another.

The problem is that, unlike a building, you can't walk around an AI tool and look at its foundation.

"For code, it's much harder to see that. Clients can see a building. They see the structure. With code, it's 'What's the big deal? We could build this, we could build that.'"
Zevi Britz

So we asked three engineers who build AI on healthcare data every day what they would want to know before they signed. Their answers came down to six questions.

‍

TL;DR - The 6 Questions You Need to Ask

If you only have a minute, this table is 80% of the article.

| # | Ask this | ✅ Green Flag | 🚩 Red flag |
|---|---|---|---|
| 1 | Who made the decisions behind this tool? | Named engineers and clinical/operational experts (DONs, administrators) who defined how every metric is calculated | "The AI figured it out." No one can explain how "census" or "readmission" is defined |
| 2 | Where does the data come from, and who keeps it clean and connected? | Automated connections to your EHR, payroll, and billing systems, a defined cleaning process, monitoring for broken or changed feeds, and experience with your specific systems | Manual spreadsheet uploads, "we can connect to your data in a weekend," no plan for new systems or acquisitions |
| 3 | Will I get the same answer today and in three months? | Metrics are pre-defined and calculated once. The AI asks for the answer instead of working it out itself every time | The AI is pointed at your raw tables and computes numbers on the fly |
| 4 | When a number looks wrong, can you show me where it came from? | A full audit trail of every step the AI took, and testing (evals) that runs before every change | "That's just how AI works." The tool is treated as a black box |
| 5 | What stops the AI from showing my nurse the CEO's salary? | Permissions enforced in the data itself. The AI only ever has the same access as the person asking. A BAA covers every system that touches PHI | "We instruct the AI not to share that." Security that lives only in the prompt |
| 6 | Who owns and maintains this on day 400? | A team that patches security updates, has been pen tested, and can describe a real issue they fixed | A one- or two-person shop, or a tool running on someone's laptop or AI subscription |


‍

About the Engineers

Srulie Vinagray

AI Lead

Zevi Britz

Senior Software Engineer

Ellie Bell

Forward Deployed Engineer

‍

Why a Great Demo Can't Tell You What's Underneath

Before getting into the questions, it helps to understand what has actually changed. AI didn't make software easy to build well. It made software easy to build.

"The coding is not the problem. The machines can write code as well as 95% of engineers. What makes an engineer an engineer is not the code that they write. It's trade-offs and decisions."
Srulie Vinagray, AI Lead, Megadata

That distinction matters, because "vibe coding" (building software by prompting AI rather than designing it) isn't automatically bad. Our engineers use AI to write code every day. The difference is whether someone with real expertise is steering.

"There are vibe-coded tools, and then there are vibe-coded tools where the vibe coder is an experienced engineer with access to reliable sources. Vibe coding can be on reliable footing. You just really have to focus on the cornerstones."
Ellie Bell, Forward Deployed Engineer, Megadata

Ellie used a term from the cybersecurity world: the script kiddie, someone who copies and pastes code without understanding how it works. AI has made it possible for anyone to build at that level. As she put it, "being a script kiddie who cares about what they're building and wants to do the best is where the difference is."

So the question isn't "Was AI used to build this?" Almost everything will be. The question is "What's holding it up?"

‍

Question 1: Who Made the Decisions Behind This Tool?

Why it matters

Every AI tool in skilled nursing makes hundreds of judgment calls you never see. What counts as a readmission? Which census number does it use? How far back does it look? Those calls determine whether the answer you get is useful, and AI can't make them on its own.

Srulie gave an example every operator will recognize:

"There's a census number in the financials, there's a census number in billing, there's a census number in the census, and there's a census number in the MDS. If I asked you for census and you gave me the billing number, you're not wrong. You just picked the wrong one. It's not factually incorrect, but you didn't get the right answer. So who's making those decisions?"
Srulie Vinagray

What the engineers said

The right decision-makers are people who understand both the technology and the work of running a building. "If you have somebody who can answer the technical questions and somebody who can answer the subject matter questions, let the AI write the code. Go for it," Srulie said. "The code is not what makes it vibe coded. It's who's the decision maker."

Zevi described the same principle from the build side. On his team, every code repository now includes written standards that tell the AI what each layer of the system is responsible for, so the AI follows the team's architecture instead of improvising a new one each time. Someone has to know enough to write those rules.

"You have to guide it. You just can't let it run loose. You have to make the decisions for it and make sure it's not taking you over."
Zevi Britz, paraphrasing advice from a former Microsoft engineer the team consults with

✅ What a good answer sounds like

"Our metric definitions were built with DONs and administrators. Here's how we define readmission, and here's who decided that." You should hear names, roles, and real-world operational input.

🚩 Red flags

• Nobody can tell you how a core metric is calculated
• The founding team has no healthcare operations or engineering background ("I read about it online" is not an answer)
• Everything is attributed to "the AI"

‍

Question 2: Where Does the Data Come From, and Who Keeps It Clean and Connected?

Why it matters

An AI tool is only as good as the data underneath it, and in skilled nursing that data is scattered. It lives in the EHR, payroll and time clock systems, scheduling, billing, staffing agency invoices, and MDS submissions. Each system has its own format, its own IDs, and its own quirks, and every operator configures them differently.

Before an AI can answer a single question, all of that has to be pulled in, cleaned, matched up, and kept current. None of that work shows up in a demo. It's also where most of the effort, and most of the failures, actually live.

"We have a team of people bringing in new companies' data, and that still takes time to make it nice for AI to talk to. And they're experienced. So for a company to think that they can just do it on a weekend..."
Ellie Bell

What the engineers said

Structure determines what the AI can find. Ellie explained it with a receipt analogy:

"If I took my shoebox of receipts from the year, threw it at you and said 'Find me my receipt where I spent $12.99,' you'd find the first one and say 'Here.' But if everything was organized, you'd say 'Okay, there are 10. This one was at Home Depot. Which one do you want?' That's really the difference between good data and unstructured data."
Ellie Bell

An AI working with a shoebox returns the first plausible answer it finds. An AI working on organized data knows what all the options are.

What "clean" actually means. In a well-built system, every data point passes through a series of steps before it ever reaches an answer:

Skip any of these, and the AI on top will still produce confident answers. They just won't be right.

Structure affects speed and cost, not just accuracy. Zevi pointed out that poorly structured data slows everything down. "If it wasn't structured properly, then querying the data is going to take longer. All that is going to affect the user." Ellie added that inexperienced builders often run AI over every step of the pipeline, even where a simple, conventional script would do the job better and cheaper. Knowing "where the line is drawn between where AI is needed and where a normal script can do it" is one of the clearest marks of an experienced engineer, and it keeps AI usage costs from ballooning.

Integrations are never "done." Source systems update, rename fields, change export formats, and get replaced. A feed that breaks loudly is an inconvenience. A feed that changes quietly, so the numbers are slightly wrong for weeks before anyone notices, is how a team stops trusting a tool.

Growth tests the architecture. Skilled nursing looks uniform from the outside, but every organization runs differently. Zevi described how a fixed portfolio-region-facility hierarchy becomes a problem when one operator wants a different structure, or when a region moves to a new portfolio and permissions get confused. The tempting response is a quick patch for each client.

"You could hack together a fix, and then the next client has a different issue. But if you sit down and think about the broader picture, you don't have to hack anything. You can build it the right way."
Zevi Britz

The same goes for acquisitions. When you add a building on a different EHR or payroll system, a well-built platform has likely connected to that system before. A tool built around one operator's setup may need to be rebuilt.

✅ What a good answer sounds like

"We connect directly to your EHR, payroll, and billing systems through automated feeds, and we've done it for many operators on those same systems. Here's what happens to your in our data warehouse between your system and the answer. Here's how we know when a feed breaks or changes. And here's a realistic timeline for getting your data ready."

🚩 Red flags

• Manual data uploads ("just export your spreadsheet and drop it in here"), which invite human error and break the first time someone forgets
• "We can connect to your data in a weekend"
• They can't name which EHR and payroll systems they've integrated with before
• No process for catching a feed that breaks or quietly changes
• No answer for what happens when you acquire a building on a different system or reorganize your regions

‍

‍

Question 3: Will I Get the Same Answer Today and in Three Months?

Why it matters

This was Srulie's first choice for the single most important question. Even with clean data underneath, an AI tool can still give inconsistent answers if it's left to decide how to calculate things on its own.

Srulie borrowed two terms from research: reliability (you get the same answer every time) and validity (the answer is actually right). "A broken clock is reliable," he said. "It's not valid." A good AI tool needs both. A tool that's consistently off by three can at least be corrected. A tool that gives a different answer every time can't be trusted at all.

What the engineers said

The difference comes down to whether the tool asks for answers or works them out from scratch.

"We already know the answer to how many falls there were in my facility this month. The AI just has to ask for that information, not go find it itself."
Srulie Vinagray

When an AI is pointed at raw database tables, it has to "SQL its way around until it finds the falls table and figure out what the denominator needs to be," as Srulie put it. It has to reinvent the calculation every time. Today it might use one facility's census as the denominator. Tomorrow it might use all facilities. Both are plausible. Only one is right.

When metrics are defined once, by the experts from Question 1, on the clean data from Question 2, the AI's job gets much simpler: understand the question and retrieve the right measure.

✅ What a good answer sounds like

"Every metric is defined once, in one place, and calculated the same way for every user. The AI retrieves those defined measures. It doesn't write its own math."

🚩 Red flags

• "We give the AI access to your database and it figures out the answer"
• The vendor can't explain how two people asking the same question get the same number
• Different users or facilities get different results for the same metric

‍

Question 4: When a Number Looks Wrong, Can You Show Me Where It Came From?

Why it matters

Every AI tool will get something wrong eventually. What separates a trustworthy tool from a risky one is what happens next.

Srulie shared a real example from his own team's work. Early in development of MegaEdge, a client asked for their Medicare average length of stay and got back a number that was obviously too high, around 123 days. It looked like the AI had invented it.

"I looked. It didn't make up a number. It just used the wrong denominator. It did everybody, all payers, and labeled it Medicare. I was able to go look and tell them, 'I'm sorry, it misinterpreted that.' That was a bug, and we fixed it."
Srulie Vinagray

The tool was wrong. But because there was a complete record of every step the AI took, the team could explain exactly why, correct it, and keep the client's trust.

"Your answer can't be 'Oh, it's AI, it does things like that.' You have to be able to answer for it. You can't treat it as a black box."
Srulie Vinagray

What the engineers said

Two things make this possible: audit trails and evaluations ("evals").

An audit trail records every move the AI makes: what it understood the question to be, what data it requested, and what came back. Evals are the testing system that checks whether answers are right, and whether a change made on a Tuesday quietly made the tool worse.

"You have to have the evals before you have the product. You have to know what's going to decide if it was good, bad, correct, incorrect before you actually build the thing that's going to be correct or incorrect."
Srulie Vinagray

One simple test Srulie described: every number in the AI's written answer must appear in the data it actually retrieved. "If it says your census today is 72, I'd better see 72 in the data response. If it's not there, the number is made up."

He was also candid about the limits. Forecasting questions ("What will my census be tomorrow?") are much harder to verify, because the answer isn't sitting in the data. A trustworthy vendor will tell you where those limits are.

Ellie added a principle that applies to every AI tool: "Autonomy has to be earned." Your team will test the waters before relying on any tool, and a good one shows its sources so they can check its work.

✅ What a good answer sounds like

"We log every step the AI takes, so we can trace any answer back to the source data. Here's our evaluation process, and here's how we test before every release." Bonus points if they share a real example of a wrong answer they caught and fixed.

🚩 Red flags

• Numbers appear with no way to trace them
• The tool never shows where its information came from
• The vendor has no testing process, or can't describe one
• The vendor claims 100% accuracy

‍

Question 5: What Stops the AI From Showing My Nurse the CEO's Salary?

Why it matters

Srulie's framing is the sharpest line in this article:

"You could have one wrong answer. You cannot have one data leak."
Srulie Vinagray

In healthcare the stakes are higher on every front. As Ellie put it: "If you pull up the wrong receipt, that's fine. If you tell me to administer the wrong amount of medication..." Add PHI, HIPAA, and employee data, and security stops being a feature. It's the foundation.

What the engineers said

Security has to live below the AI, not inside its instructions. Many tools protect sensitive data by telling the AI not to share it. The problem is that instructions can be talked around. Ellie explained that "jailbreaking" an AI works a lot like getting a person to reveal a secret: "You manipulate them in the right way and they tell you the secret. AI is no different."

"The security needs to be above the prompt level. Prompting it, 'You only get to see this user's data,' could be prompted around. You need to go at the foundational level, so the AI can never have access to it, even if it went rogue and tried to get it."
Srulie Vinagray

The fix is role-based access control (RBAC) and row-level security (RLS) enforced in the data itself. The AI only ever operates with the permissions of the person asking. "The AI doesn't get to choose whose permissions it has," Srulie said. "It's just an extension of the user. Not more powerful." Ellie put it more simply: with proper permissions, the AI "doesn't even know anyone else" exists.

Every system that touches PHI needs to be covered. Srulie listed the pieces an AI tool might use, each of which needs a Business Associate Agreement (BAA) if PHI passes through it:

Leaks are often invisible to users. Zevi described seeing a vendor's product where anyone could open their browser's developer tools and see what time another company's employees clocked in that day. Nothing on screen gave it away. That's why independent penetration testing (hiring security professionals to try to break in) matters so much.

✅ What a good answer sounds like

Ellie suggested starting with "Is it HIPAA compliant?", but warned that a simple "yes" isn't enough. "HIPAA compliance requires a specific type of security, specific hosting, and specific ways to reach the data. A good answer is yes, followed by an explanation that proves they know what they're talking about." Listen for: a SOC 2 report, a BAA with their cloud host and AI provider, permissions enforced at the data level, and recent pen test results.

🚩 Red flags

• "We tell the AI not to share that"
• You can ask the tool about facilities or people you shouldn't have access to and get an answer. (Ellie: "Then it's probably giving other people my data.")
• No BAA with the AI model provider
• The tool has never been pen tested
• A simple "yes, we're HIPAA compliant" with nothing behind it

‍

Question 6: Who Owns and Maintains This on Day 400?

Why it matters

Buying an AI tool isn't a one-time purchase. As Zevi pointed out, "You're not just paying a monthly subscription. You're changing how you run your operation. You're investing in this person's tool."

Software doesn't stand still after launch, especially software handling PHI. Zevi gave a timely example: Microsoft ends support for .NET 8, a widely used software framework, in November 2026. After that date, new security vulnerabilities won't be patched. Every product built on it needs an upgrade, and upgrades cascade. When Google required apps to support the latest Android version, his team's mobile app update forced an upgrade of the entire underlying framework.

"Right now it's fun. Everyone can spin up their own apps. Okay, I got it working. Fast forward a couple of months. There are always updates, and a lot of times updates are for security reasons. Who's following that? Who's updating the mini apps that you made?"
Zevi Britz

AI made that upgrade faster and the testing better, Zevi said, "but you still have to watch, see what the breaking changes are, and then make decisions."

What the engineers said

Where the tool runs matters. Ellie described how easy it is to build a tool that runs on one person's laptop and share it over the office network. "The moment I shut my computer or leave the Wi-Fi, you can't access it." Other tools run entirely on someone's AI subscription. "The moment they run out of usage, the software doesn't work anymore for five hours."

Small teams carry real risk. Zevi raised the "bus factor": if the tool depends on one or two people, what happens if they leave? "Is the hype going to be gone, and in two years they're selling shoes?"

Ellie added a less obvious benefit of working with a vendor that serves many facilities: they learn from everyone. A feature one operator asks for often turns out to be something others wanted too. Building alone, "you would never get that if you're confined by the silo of your own thinking."

✅ What a good answer sounds like

Zevi's one question for any vendor was about ownership: do they truly own this product, and will they be there when something goes wrong? Ask for proof.

"A real company is going to have to own it and take responsibility. They'll have to be up at 2 o'clock, 3 o'clock to figure it out when it happens."
Zevi Britz

Ask them to walk you through a real technical issue a client had and how they resolved it. Ask about uptime, response times, and their security update process.

Red flags

• A one- or two-person team with no plan for continuity
• The tool runs on a personal computer or a consumer AI subscription
• Manual data uploads ("just export your spreadsheet and drop it in here"), which invite human error and break as soon as someone forgets
• No answer for what happens when you add buildings, systems, or a new org structure
• No history of handling a security issue

‍

Before You Sign: The Six Questions, Side by Side

Print this, bring it to your next vendor meeting, and listen for specifics. Vague answers are answers too.

1. Who made the decisions behind this tool? Listen for named experts and defined metrics.
2. Will I get the same answer today and in three months? Listen for pre-defined measures the AI retrieves rather than recalculates.
3. When a number looks wrong, can you show me where it came from? Listen for audit trails, evals, and a real story about a bug they fixed.
4. What stops the AI from showing my nurse the CEO's salary? Listen for permissions enforced in the data, BAAs across the stack, SOC 2, and pen testing.
5. Who owns and maintains this on day 400? Listen for a team, a track record, and a plan for growth.

‍

Frequently Asked Questions

Is "vibe coding" always a red flag?

No. Most modern software is written with AI help, including ours. The red flag is when nobody with engineering and healthcare expertise is making the decisions. As Srulie put it, "The code is not what makes it vibe coded. It's who's the decision maker."

How long should it take to connect an AI tool to our data?

Longer than a demo suggests. Getting data from multiple skilled nursing systems clean and consistent enough for AI to rely on takes experienced teams weeks to months. A vendor promising a weekend is either skipping steps or underestimating your data.

Isn't "HIPAA compliant" enough?

It's a starting point, not an answer. Ask how they're compliant: who they have BAAs with, where data is hosted, how permissions are enforced, and whether they have a SOC 2 report. See Megadata's SOC 2 and HIPAA compliance.

What's the difference between an AI being told not to share data and not having access to it?

Instructions can be talked around (often called "jailbreaking"). If the AI never has access to data beyond the user's permissions, no phrasing can get it out.

How can a non-technical operator test an AI tool during a trial?

Ask it about facilities or people you shouldn't have access to. Ask the same question on different days and compare the answers. Ask it where a number came from. Then ask the vendor to explain one wrong answer.

What is an "eval"?

An evaluation system that checks whether an AI tool's answers are correct, and whether changes to the tool made it better or worse, before those changes reach users.

The Bottom Line

The new generation of AI tools is real, and many of them will make skilled nursing operations faster and smarter. But the part that makes a tool trustworthy is the part you can't see in a demo: the definitions, the data and integrations underneath, the permissions, and the people who will still be maintaining it next year.

So before you move your team onto the top floor, ask what's holding the building up.

These six questions are the same standards our engineers hold themselves to while building MegaEdge, Megadata's AI layer for healthcare operations.

About the Engineers

Srulie Vinagray, AI Lead, Megadata.

Ellie Bell, Forward Deployed Engineer, Megadata.

Zevi Britz, Senior Software Engineer, Megadata.