Go to Top

Hollowing Out The IRS

red, white and blue M&M's with custom messages reading Yay Audits! and Audits Rock! against a white tablecloth
Probably not enough to make an IRS employee popular. Photo by Joe Hall.

Like most people, I don’t exactly jump for joy when I hear from the Internal Revenue Service, but I have to feel a little sorry for many of the people who work there.

Practically everyone avoids them at cocktail parties. The pay isn’t great and the facilities are terrible. Usually, when an IRS auditor wants to examine a return filed by one of my clients, the first thing I have to do is provide the return - the agency’s systems can’t even show the auditors exactly what they are auditing. On top of that, the laws that the Service must enforce are fiendishly complicated, and many field-level agents are only scantily trained. Then there are the shrinking ranks of customer service representatives, whose entire day is devoted to fixing, or at least trying to fix, problems that are never of their own making. They succeed only occasionally.

When the rest of us talk about the IRS, we usually do so either with loathing and scorn or as the butt of jokes. The IRS and its agents are popular targets in entertainment, as in a memorable scene from the 2006 comedy, “Stranger Than Fiction.” They are also easy targets in Congress, where lawmakers, especially Republicans, have little sympathy. “We deliberately lowered the IRS funding to a level to make them think twice about what they are doing and why they are doing it,” Rep. Ander Crenshaw, R-Fla., said in a hearing in February.

Granted, many of the IRS’ problems are self-inflicted. There have been horror stories for decades about IRS misanthropes who take every opportunity to threaten and bully taxpayers. The scandal that led to Lois Lerner’s retirement, in which the IRS singled out Tea Party and other conservative groups applying for nonprofit status for extra scrutiny, gave the Service a partisan sheen it has yet to shake. Tax authorities have recently blamed the public for the uptick in tax and identity fraud caused by electronic filing - which was the authorities’ idea in the first place.

The agency also has a bad habit of overreaching. Lawmakers have recently pushed the IRS to back off from its overzealous tendency to seize assets from cash-dependent small businesses which it suspects, but does not need to prove, have structured transactions to avoid currency reporting rules.

Yet none of this changes the fact that it is in nobody’s best interest to see the agency unable to accomplish its core mission of carrying out the tax laws. The IRS is being systematically hollowed out, to the point that it is almost unable to function. Audits of high-income households have dropped to a six-year low. Business audits in 2014 were at their lowest rate since 2005. Criminal investigations, recommended prosecutions and staff levels are all down too. Even simple administrative tasks, such as issuing taxpayer identification numbers to foreigners who need them to file U.S. returns, require special efforts at the struggling agency.

IRS Commissioner John Koskinen called the decrease in audit rates “deeply disturbing,” and made clear that cuts in funding directly translate to cuts in audits. “There are fewer audits because we have fewer auditors,” he said in prepared remarks to the New York State Bar Association. The agency is currently under a hiring freeze.

President Obama has asked for an 18 percent budget increase for the IRS, but he is not going to get it, especially while his administration continues to battle Congress over the Tea Party abuses. Congressional Republicans have argued that the IRS can pick up the slack through increased efficiency and by cutting things like bonuses and conferences. That is not true, but the bigger problem is that the people running Congress have no faith in the agency’s ability or willingness to use any resources it is given in a fair and competent manner. So the agency is, for now, doomed to get less.

The IRS should be completely restructured and refocused on administering the tax laws in an efficient and impartial way, in which the rights and obligations of taxpayers are given equal weight with the government’s need for revenue. Further, the tax code should be streamlined to provide for better administration. Repealing estate and gift taxes, an issue that is back on Congress’ radar, would be a great first step. Those taxes require a lot of enforcement resources and generate endless valuation controversy, in return for relatively little revenue. The IRS’ computer systems should be modernized and basic anti-fraud measures should be put in place.

But first, we need to get the IRS to stop doing things it ought not to be doing, including abusing taxpayers and seizing property without justification just because it can. Otherwise IRS employees will have to keep hiding their affiliation whenever they attend social gatherings. Simply providing the Service with more money won’t fix the cultural and management problems at the IRS. Without major changes, the agency and the country can both expect many lean years ahead.

Larry M. Elkin is the founder and president of Palisades Hudson, and is based out of Palisades Hudson’s Fort Lauderdale, Florida headquarters. He wrote several of the chapters in the firm’s recently updated book, Looking Ahead: Life, Family, Wealth and Business After 55. His contributions include Chapter 1, “Looking Ahead When Youth Is Behind Us,” and Chapter 4, “The Family Business.” Larry was also among the authors of the firm’s book The High Achiever’s Guide To Wealth.

The views expressed in this post are solely those of the author. We welcome additional perspectives in our comments section as long as they are on topic, civil in tone and signed with the writer's full name. All comments will be reviewed by our moderator prior to publication.

, , , , , ,

3 Responses to "Hollowing Out The IRS"

  • Kent Land
    April 15, 2015 - 9:23 am

    keep hollowing out the irs, until there is nothing there to hollow out. maybe they will get real jobs.

  • Norm Glitz
    April 15, 2015 - 9:31 am

    Once you get to your last paragraphs, it sounds like you agree wholeheartedly with tthe Republican assessment that increased efficiency is the way to go.

    Conferences are typically a waste of time; vacation on the government’s (taxpayers’) tab. Doing thigs they ought not to be doing is a huge waste of everything, except for lawyers getting paid, also on the taxpayer’s tab. Wasting time on “endless valuation controversy” is another huge inefficiency. And yes, the tax code needs to be greatly simplified, but that’s on congress. Good luck there.

    Maybe we should try congress’ own medicine on congress: “We deliberately lowered the IRS funding to a level to make them think twice about what they are doing and why they are doing it ” Congressmen should be eliminating old, useless laws and writing newer more efficient laws, not writing generalized laws and letting the bureaucracies write “Regulations” that they enforce as law. Instead, they’re getting rich on lobbyists and getting re-elected.

    Now there’s an idea whose time has come. More good luck needed.

  • Kendra Kylie
    April 15, 2015 - 1:33 pm

    Time for a flat rate sales tax, across the board – equitable and applicable to everyone. Have it collected on the gross amount all sales, services and merchandise by enterprises and businesses and remitted at the county level. It should first go to pay for local services, with a small percentage of the collected tax allocated to the State, then the Federal governments for services. The more money a person spends, the more tax they pay. The only type of entity I could envision as exempt from collecting and remitting sales tax would be an individual doing a garage sale.