Two leaders of the House Freedom Caucus warn they won’t support a “bad immigration bill.”

As is traditional during Infrastructure Week, nobody on Capitol Hill is talking about infrastructure. Instead, lawmakers are zeroing in on immigration and the young undocumented immigrants whose protection from deportation expires in March.

But while all eyes are on the Senate’s not-so-freewheeling debate on those issues, the real action is perhaps in the lower chamber. There, House Republicans are grappling with their own path forward on immigration—and on Wednesday morning, internecine tensions boiled over. Representative Mark Meadows, chairman of the House Freedom Caucus, offered a direct challenge to House Speaker Paul Ryan: “I can say that it is the defining moment for this speaker,” he said during Conversations with Conservatives, a monthly press conference with members of the House’s right flank. “If he gets it wrong it will have consequences for him, but it will also have consequences for the rest of the Republican Party.” (Ben Williamson, Meadows’s press secretary, quickly tweeted that Meadows was not asking for a personnel change.)

As House leadership quietly awaits action in the Senate, conservatives are anxious to move ahead with their own immigration legislation. House Freedom Caucus members and others have rallied around a bill, authored by Judiciary Committee Chairman Bob Goodlatte, that would end family-based migration; slash the visa lottery program; authorize funds for a border wall; and offer renewable legal status for DACA recipients instead of a path to citizenship, as the White House proposal does.

The bill, which Meadows estimated had 160 to 180 “yeas” as of Tuesday morning, is likely dead-on-arrival in the Senate. But House conservatives say they are more determined than ever to get the bill to the floor, after the massive bipartisan budget deal that passed last week struck a “blow” to the Freedom Caucus, as Meadows put it. In a joint interview Tuesday afternoon, Meadows and his predecessor, Ohio Representative Jim Jordan, suggested that Paul Ryan’s speakership could potentially be on the line if leadership fails to introduce Goodlatte’s legislation and muscles through a “liberal” bill from the Senate.

Meadows told me Ryan won’t dare consider the Senate’s legislation “if he knows the will of his conference. The will of the conference would not be supporting that.” The budget deal may have passed the House with Democratic support, Meadows added, “but the last time I checked, he got elected to be Speaker for Republicans.”

Perhaps no issue induces political tremors quite like immigration, especially in recent years. “I defy you to find worse debates in recent history than those over immigration,” veteran Democratic Senate aide Jim Manley told my colleague Russell Berman this week, adding that “they are ugly, bloody debates chock-full of highly partisan social issues.” But with a deadline looming on DACA—coupled with a president who was in many ways swept into the White House because of his immigration policies—lawmakers are again forced to confront the politically fraught topic. Bipartisan and bicameral struggles alike may ensure that, once again, the debate ends in a stalemate. But that doesn’t mean lawmakers want to avoid the issue: In the eyes of conservatives and Democrats alike, House leadership appears alarmingly comfortable with the potential of passing no legislation at all.

“I think the most frustrating thing about this is just how weak Ryan has become, in that he views the issue as his potential downfall, so he won’t act,” an aide to House Minority Leader Nancy Pelosi told me on Tuesday. “Put Goodlatte on the floor, as terrible as it is. Put whatever comes from the Senate on the floor. Let’s just have a vote.”

Conservatives believe House leadership has lacked enthusiastic follow-through on a deal they negotiated three weeks ago, when conservatives gave their votes for a stopgap government-funding bill in exchange for leadership’s pledge to whip support for the Goodlatte legislation. “I’m worried, because we were promised a full-scale push and whip operation on Goodlatte and we’ve just frankly not seen anything close to that,” Jordan said in the joint interview. “It’s basically Bob Goodlatte going around and talking to people, and the Freedom Caucus going around and talking to people. Other than that, you don’t hear anything from leadership about the legislation.”