Order sets are foundational to clinical workflows, but they routinely fall behind evolving evidence. We built a framework to close that gap, and we're doing a systems demo at AMIA on May 20th to discuss how we're tackling this challenge.

When was the last time your team reviewed every order set in your EHR? If the honest answer is "it's been a while," you're not alone. The problem isn't effort. It's scale.

Clinical evidence moves fast. Guidelines get updated, best practices shift, and contraindications get reclassified. Order sets, meanwhile, tend to stay where they were , creating a slow, invisible drift between what the evidence says and what your workflows do.

Most health systems rely on periodic manual review to keep order sets current. But the volume is enormous, the process is labor-intensive, and it's nearly impossible to know what you don't know. By the time a discrepancy surfaces, it may have been sitting in a workflow for months.

What's needed is a way to continuously monitor concordance between external clinical knowledge and internal EHR content at scale — without requiring a team of humans to read every guideline update.

We’re developed an LLM-based framework that automates three core steps:

• Automatically linking external clinical knowledge artifacts (guidelines, evidence sources) to the relevant local EHR order sets
• Identifying differences between those linked entities: flagging what's missing, what's outdated, and what may actively conflict with current guidance
• Enabling proactive, targeted content reviews without the heavy manual lift

We deployed the framework at two pilot health systems. Seven clinically meaningful discrepancies were identified — falling into two categories:

1. ‍Omission: Missing new evidence-based recommendations. The neonatal fever order set lacked orders for inflammatory markers that current evidence now recommends.
2. Commission: Guidance that no longer aligns with current evidence. A vaccine order set instructed providers to discuss egg allergies with an attending, a precaution that the updated evidence no longer supports.

These aren't edge cases engineered to make the system look good. They're the kind of real-world discrepancies that are easy to miss and hard to find without a system actively doing the looking.

Future work on our Knowledge Sync product will expand evidence source coverage, enhance temporal tracking, and evaluate impact on clinical outcomes. If you're looking to keep EHR content aligned with rapidly changing evidence, without the manual lift, we'd love to tell you more.