Having a government-recognized marriage entitles a couple to federally funded benefits, but as the government is a public institution, that means that documentation of the marriage is available to the public. There's no way around that as far as the U.S. government is concerned. A formal request for the information is required and it is up to the Federal Information Center to provide it if it is a valid reason. This can deter most of the public from seeing your marriage certificate, but if you want your marriage completely off the public record, consider alternatives to legal marriage.
Skip the legal formalities altogether. There is no law saying that you and your partner have to sign documents in order to have a commitment. However, this could cost you many legal benefits that come with a recognized marriage. For example, if one of you is hospitalized, the other may not have visitation rights. In the event one of you passes away without a will, the other may not have automatic inheritance. Private insurance policies may not give you joint benefits. If you decide to split up, the lack of a license can save you cost of lawyers and time spent with paperwork, but you may not be able to claim alimony or child support.
Register for domestic partnership. This lets you have some of the benefits of a licensed marriage without having it legally documented. This is an option for same-sex and opposite-sex couples. Some employers' insurance policies honor domestic partnership for both no-cost benefits and sick leave for one partner to care for the other, while some private insurers offer equal benefits to married couples and domestic partners. AAA offers a reduced fee for "second drivers" that was originally only for legal spouses. Overall, domestic partnership is not equal to legal marriage. The benefits of domestic partnership vary greatly from state to state and can be amended or revoked with just a change in legislation.
Apply for common-law marriage. Common-law marriage does not require a license, but it does grant the legal benefits of marriage provided that the couple has lived together for a certain amount of time, follow other marriage customs such as referring to each other as "husband" and "wife" or using the same last name, and show intent to be legally married. Common-law marriage requirements vary by state and is only offered in 12 states along with American Samoa.
Research marriage and domestic partnership laws in another country. You can then apply for dual citizenship, register for a marriage license, and reap that country's benefits. The U.S. government is not required to recognize marriage licenses from other countries, but according to the Gay and Lesbian Partners Task Force, it usually does unless the marriage violates public policy somehow. That "somehow" can apply to same-sex marriages in states where such marriages are not legal.