Inclusion Criteria:

* PRIOR TO STEP 1 REGISTRATION:

* Brain metastases outside a 5-mm margin around either hippocampus must be visible on contrast-enhanced magnetic resonance imaging (MRI) performed =\< 21 days prior to Step 1 registration; an allowed exception, regarding ability to image brain metastases, would be that patients who had undergone radiosurgery or surgical resection and are planning adjuvant WBRT do not have to have visible disease but do need a pre-surgery MRI or computed tomography (CT) scan demonstrating brain metastases; however, the brain metastases could not have been within 5 mm of either hippocampus
* Patients must have a gadolinium contrast-enhanced three-dimensional spoiled gradient (SPGR), magnetization-prepared rapid gradient echo (MP-RAGE), or turbo field echo (TFE) axial MRI scan with standard axial and coronal gadolinium contrast-enhanced T1-weighted sequence and axial T2/FLAIR sequence acquisitions; to yield acceptable image quality, the gadolinium contrast-enhanced three-dimensional SPGR, MP-RAGE, or TFE axial MRI scan should use the smallest possible axial slice thickness not exceeding 1.5 mm; the associated coronal and sagittal contrast-enhanced T1 sequences can be up to 2.5 mm in slice thickness; this MRI must be obtained =\< 21 days prior to step 1 registration; the vendor specific MRI protocols are available for download from the Alzheimer's Disease Neuroimaging Initiative (ADNI)
* Patients must provide study-specific informed consent prior to registration
* PRIOR TO STEP 2 REGISTRATION:

* The following baseline neurocognitive assessments must be completed prior to Step 2 registration: HVLT-R, TMT, and COWA;
* Pathologically (histologically or cytologically) proven diagnosis of solid tumor malignancy within 5 years prior to Step 2 registration
* History and physical examination within 28 days prior to Step 2 registration
* Karnofsky performance status of \>= 70 within 28 days prior to Step 2 registration
* Serum creatinine =\< 3 mg/dL (265 umol/L) and creatinine clearance \>= 30 ml/min
* Blood urea nitrogen (BUN) within institutional upper limit of normal (e.g. \< 20 mg/dL)
* Total bilirubin =\< 2.5 mg/dL (43 umol/L)
* Patients may have had prior therapy for brain metastasis, including radiosurgery and surgical resection; patients must have completed prior therapy by at least 14 days prior to Step 2 for surgical resection and 7 days for radiosurgery
* Negative serum pregnancy test (in women of childbearing potential) =\< 14 days prior to Step 2; women of childbearing potential and men who are sexually active must practice adequate contraception while on study
* Patients who are primary English or French speakers are eligible

Exclusion Criteria:

* Prior external beam radiation therapy to the brain or whole brain radiation therapy
* Planned cytotoxic chemotherapy during the WBRT only; patients may have had prior chemotherapy
* Radiographic evidence of hydrocephalus or other architectural distortion of the ventricular system, including placement of external ventricular drain or ventriculoperitoneal shunt
* Severe, active co-morbidity defined as follows:

* Unstable angina and/or congestive heart failure requiring hospitalization within the last 6 months
* Transmural myocardial infarction within the last 6 months
* Acute bacterial or fungal infection requiring intravenous antibiotics at the time of registration
* Chronic obstructive pulmonary disease exacerbation or other acute respiratory illness precluding study therapy at the time of registration
* Severe hepatic disease defined as a diagnosis of Child-Pugh class B or C hepatic disease
* Renal tubular acidosis or metabolic acidosis
* Human immunodeficiency virus (HIV) positive with cluster of differentiation (CD)4 count \< 200 cells/microliter; note that patients who are HIV positive are eligible, provided they are under treatment with highly active antiretroviral therapy (HAART) and have a CD4 count \>= 200 cells/microliter within 30 days prior to registration; Note also that HIV testing is not required for eligibility for this protocol
* Pregnant or lactating women, or women of childbearing potential and men who are sexually active and not willing/able to use medically acceptable forms of contraception
* Prior allergic reaction to memantine (memantine hydrochloride)
* Current alcohol or drug abuse (may exacerbate lethargy/dizziness with memantine)
* Intractable seizures while on adequate anticonvulsant therapy-more than 1 seizure per month for the past 2 months
* Patients with definitive leptomeningeal metastases
* Patients with brain metastases from primary germ cell tumors, small cell carcinoma, unknown primary, or lymphoma
* Contraindication to magnetic resonance (MR) imaging such as implanted metal devices or foreign bodies
* Contraindication to gadolinium contrast administration during MR imaging, such as allergy or insufficient renal function
* Current use of (other N-methyl D-aspartate \[NMDA\] antagonists) amantadine, ketamine, or dextromethorphan